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results  of  a one-year  research  and  development  program  aimed  at  bringing  these 
two  areas  together  in  order  to  define  the  decision  training  objectives  of 
typical  military  systems  and  to  specify  ways  of  meeting  them  in  terms  of  a set  ofj 
guidelines  for  neneration  of  instructional  materials  for  decision  training  systemp 

The  program  is  divided  into  four  phases:  (1)  Job  Analysis  Phase,  (2)  Decision 
Task  Analysis  Phase,  (3)  Development  and  Evaluation  Phase,  and  (4)  Documentation 
Phase.  In  the  first  phase,  the  SH-2F  Light  Airborne  Multi-Purpose  System  (LAMPS) 
helicopter  training  system  was  analyzed.  Based  on  this  analysis,  the  decision 
situations  that  occur  during  emergency  and  tactical  system  operations  were 
identified,  and  a number  of  decision  tasks  required  to  resolve  a selected  set  of 
decision  situations  were  chosen.  These  decision  tasks  were  the  input  to  the 
second  phase  where  they  were  analyzed  to  identify  the  required  facts,  the  under- 
lying concepts,  the  decision  rules,  and  the  step-by-step  procedure  that  determine} 
the  optimal  choice  of  al ternati ves.  Based  on  the  results  of  this  analysis, 
instructional  guidelines  were  produced  in  the  third  phase.  An  evaluation  plan 
was  developed  and  decision  aid  systems,  to  enhance  the  effect  of  the  decision 
training  program,  were  suggested. 

During  the  one-year  program  a system  to  generate  instructional  guidelines  for  a 
task-specific  decision  training  program,  in  any  training  environment,  was 
developed.  The  system  selects  the  most  effective  instructional  method  and  media 
and  provides  required  instructional  contents  for  any  specific  decision  task, 
subject  to  training  in  a specific  training  environment.  The  project  has  been 
aimed  not  only  to  develop  a systematic  procedure  for  providing  instructional 
guidelines  for  decision  tasks  encountered  in  LAMPS  Anti-Submarine  Warfare  (ASW) 
operation,  but  also  to  design  a task- independent  methodology  in  order  to  make 
application  of  the  system  in  other  task  domains  subject  to  minimal  modification 
effort.  Such  a methodology  allows  incorporation  of  general  purpose  decision 
training  components  into  task  specific  decision  training  programs.  This 
incorporation  provides  trainees  with  the  basic  skills  required  for  making  task- 
specific  decisions  within  a new  domain,  in  case  of  their  transfer  to  other 
domains.  Furthermore,  since  the  characteristics  of  decision  tasks  within  any 
specific  domain  are  subject  to  change  with  time,  the  methodology  will  allow 
development  of  more  effective  training  programs  even  in  the  case  where  there  is 
no  possibility  of  transferring  the  trainees  to  other  task  domains. 

The  program  concentrated  on  two  major  areas  in  the  Navy  LAMPS  (ASW)  Operation: 
emergency  operations  and  tactical  operations.  The  methodology  for  both  areas 
as  well  as  the  specific  activities  for  the  emergency  operations  are  included 
in  this  report.  The  results  of  the  specific  activities  for  the  tactical 
operations  are  described  in  a separate  classified  supplement. 
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SUMMARY 


This  report  presents  the  results  of  a one-year  research  and  development 
program  in  the  area  of  decision  training.  The  project  covers  two  areas  of 
the  Navy  SF-2F  Light  Airborne  Multi-Purpose  System  (LAMPS)  Anti-Submarine 
Warfare  (ASW)  operation:  emergency  and  tactical  operations.  This  report 
describes  the  unclassified  results  of  the  project.  Other  materials  have 
been  presented  in  a separate,  classified  report. 

This  report  is  organized  into  five  chapters.  The  first  chapter  is  an 
introduction  which  includes  an  overview  of  the  project,  background,  and 
technical  approach.  Chapters  2,  3,  and  4 correspond  to  the  first  three 
phases  of  the  program.  The  results  of  the  job  analysis  phase  is  in  Chapter 
2.  In  this  phase,  the  tasks  involved  in  the  Navy  SH-2F  (LAMPS)  helicopter 
training  system  were  analyzed,  decision  tasks  were  identified,  and  a number 
of  the  identified  tasks  were  selected  for  further  analysis.  Chapter  3 
describes  the  second  phase  of  the  project:  decision  task  analysis.  The 
selection  decision  tasks  were  analyzed  to  identify  the  required  facts,  the 
underlying  concepts,  the  decision  rules,  and  the  step-by-step  procedures 
that  determine  the  optimal  choice  of  alternatives.  Based  on  the  results 
of  this  analysis,  a system  for  instructional  guideline  generation  was 
developed.  The  instructional  guideline  generation  system  is  presented  in 
Chapter  4.  The  system  provides  a procedure  to  generate  essential  information 
to  be  incorporated  in  the  production  of  instructional  materials  for  any 
decision  task  subject  to  training  in  any  specific  instructional  environment. 
Each  of  the  chapters  2 through  4 starts  with  a presentation  of  the  general 
approach  and  activities  associated  with  the  corresponding  phase.  The 
following  sections  cover  the  methodology  designed  and  utilized  in  the 
corresponding  phase  of  the  project.  Technical  details  and  the  results  of 
the  phase  appears  in  the  remaining  sections.  The  last  chapter  of  the  report 
presents  the  conclusions  of  the  project.  The  dimensions  for  improvement  of 
the  system  developed  during  the  project  are  identified  and  a computerized 
instructional  guideline  generated  aid,  as  a means  to  achieve  improvements 
in  all  of  these  dimensions,  is  briefly  described.  Also  included  are 
suggestions  for  decision  aids  based  on  the  results  of  the  analyses,  an 
evaluation  plan  for  the  instructional  materials  developed  for  decision  tasks, 
and  the  results  of  the  instructional  guideline  generation  system.  The  primary 
results  of  the  analysis  and  decision  trees  appear  in  the  appendices. 

During  the  one  year  project  period,  a system  to  generate  instructional 
guidelines  for  a task-specific  decision  training  program,  in  any  training 
environment,  was  developed.  The  system  selects  the  most  effective 
instructional  method  and  media  and  provides  required  instructional  contents 
for  any  specific  decision  task,  subject  to  training  in  a specific  training 
environment.  The  project  has  been  aimed  not  only  to  develop  a systematic 
procedure  for  providing  instructional  guidelines  for  decision  tasks 
encountered  in  LAMPS  ASW  operation,  but  also  to  design  a task-independent 
methodology  in  order  to  make  application  of  the  system  in  other  task 
domains  subject  to  minimal  modification  effort.  Such  a methodology  allows 
incorporation  of  general  purpose  decision  training  components  into  task 
specific  decision  training  programs.  This  incorporation  provides  trainees 
with  the  basic  skills  required  for  making  task-specific  decisions  within 
a new  domain,  in  case  of  their  transfer  to  other  task  domains.  Furthermore, 
since  the  characteristics  of  decision  tasks  within  any  specific  domain  are 
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subject  to  change  with  time,  the  methodology  will  allow  development  of  more 
effective  training  programs  even  in  the  case  where  there  is  no  possibility 
of  transferring  the  trainees  to  other  task  domains.  Finally,  the  methodology 
will  provide  for  carry  over  of  the  decision  making  capability  into  incidental 
functions,  which  are  usually  left  untrained. 
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PREFACE 


Decision  task  training  is  currently  not  receiving  required  emphasis 
in  formal  training  environments,  most  probably  due  to  a lack  of  appropriate 
training  technology.  The  training  of  decision  tasks  still  takes  place  on 
the  job,  a costly  process  in  terms  of  dollars  and  time. 

The  work  summarized  in  this  technical  report  represents  an  attempt 
to  meld  decision  theoretic  concepts  with  operational  knowledge  in  order 
to  advance  the  state-of-the-art  in  the  area  of  decision  task  training. 

It  is  anticipated  that  the  procedures  which  have  been  developed  under 
this  contract,  after  subsequent  validation  and  refinement,  will  be 
incorporated  into  the  Instructional  Systems  Development  (ISD)  process. 

Appreciation  is  expressed  to  the  following  individuals  who 
enthusiastically  participated  as  subject  matter  experts  in  their  areas  of 
expertise:  Mr.  A.  Gibbons  of  Courseware,  Inc.,  San  Diego,  and  the 
personnel  of  HSL  31,  NAS  North  Island,  San  Diego  - 
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SECTION  I 
INTRODUCTION 


OVERVIEW 

A one-year  research  and  development  program  in  the  area  of  decision 
training  is  described  in  this  report.  The  program  investigated  the  use  of 
decision  analysis  techniques  in  the  design  of  instructional  materials  for 
decision  training  in  military  systems.  The  long  range  goal  is  to  define  a 
decision  training  approach  that  can  be  applied  to  a wide  variety  of  system 
training  courses.  As  a first  step  toward  that  goal,  the  present  program 
focused  on  the  decision  training  requirements  of  the  Navy  SH-2F  Light  Airborne 
Multi-Purpose  System  (LAMPS)  Anti-Submarine  Warfare  (ASW)  helicopter.  In  this 
environment,  decision  analysis  techniques  were  used  to  (1)  identify  and 
select  critical  decision  making  situations  during  helicopter  operation,  (2) 
define  the  decision  making  tasks  required  of  the  Pilot,  Airborne  Tactical 
Officer  (ATO),  and  the  Sensor  Operator,  and  (3)  specify  the  skills  and 
knowledge  necessary  to  make  effective  decisions. 

To  an  ever-increasing  extent,  the  operators  of  advanced  military  systems, 
such  as  aircraft,  ships,  weapons,  C3,  etc.,  act  primarily  as  decision  makers. 
That  is,  the  operator's  most  important  functions  are  to  select  from  alternative 
data  sources  and  to  choose  among  alternative  courses  of  action.  Although  he 
may  be  well  trained  to  carry  out  each  alternative  procedure,  he  is  virtually 
never  trained  in  how  to  most  effectively  make  the  crucial  decision  among  them. 
Traditionally,  acquiring  the  judgment  to  make  sound  decisions  has  been 
considered  an  art,  which  must  be  developed  over  years  of  experience,  or 
through  long  apprenticeship  to  a veteran  decision  maker.  This  approach, 
aside  from  being  of  dubious  validity,  is  inappropriate  in  today's  military 
environment,  where  an  operator  must  step  into  a job  and  function  well  from 
the  start. 

Within  the  past  several  years,  the  general  rules  of  effective  decision 
making  have  been  consolidated  into  the  technical  area  termed  "decision 
analysis."  During  the  same  period,  the  systems  approach  to  training  has 
established  a generalized  methodology  for  dealing  with  diverse  content  areas 
in  the  design  of  training  courses  called  Instructional  System  Development 
(ISO).  Accordingly,  it  is  the  objective  here  to  bring  these  two  areas 
together  in  order  to  define  the  decision  training  objectives  of  typical 
military  systems  and  to  specify  ways  of  meeting  them.  Successful  joining 
of  the  two  technologies  should  have  a profound  effect  on  future  system 
performance  and  lifetime  cost. 

The  current  program  has  the  following  major  objectives: 

a.  To  analyze  a specific  military  training  system,  the  SH-2F  (LAMPS) 
helicopter,  in  terms  of  specific  decision  analysis  techniques. 

b.  To  identify  the  decision  situations  that  occur  during  regular  and 
emergency  system  operations. 
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c.  To  isolate  the  decision  tasks  that  are  required  to  resolve  a 
selected  set  of  decision  situations. 

d.  To  identify,  for  each  decision  task,  (a)  the  required  facts, 

(b)  the  underlying  concepts,  (c)  the  decision  rules,  and  (d)  the 
step-by-step  procedures  that  determine  the  optimal  choice  of 
alternatives. 

e.  To  contribute  to  the  production  of  training  materials  for  one 

or  more  of  the  decision  tasks,  to  include:  (a)  training  analysis, 
and  (b)  production  of  instruction. 

f.  To  evaluate  the  materials  produced  for  this  test  case,  and  to 
analyze  the  implications  for  (a)  similar  system-oriented  courseware, 
(b)  courseware  for  generalized  decision  training,  and  (c)  decision- 
aiding  software  for  the  SH-2F  and  related  systems. 

RATIONALE 

The  term  "decision  making"  has  been  used  to  describe  a wide  range  of 
cognitive  processes.  Building  upon  the  classification  of  others,  Nickerson 
and  Feehrer  (1975)  have  conceptualized  decision  making  as  a collection  of 
problem  solving  or  decision  tasks  which  involve:  "information  gathering, 
data  evaluation,  hypothesis  generation,  problem  structuring,  hypothesis 
evaluation,  preference  specification,  action  selection,  and  decision  evalua- 
tion." 


Study  of  human  decision  making  beha 
plished  with  the  different  decision  maki 
1970).  In  the  literature,  the  term  "def 
(1)  to  refer  to  stereotypes  ways  of  beha 
tendency  of  humans  to  be  overly  conserva 
bi  1 itistic  information  to  the  evaluation 
basic  human  cognitive  limitations  such  a 
processing  limits  such  as  the  inability 
some  small  number  of  factors  in  arriving 
without  procedural  help. 


vior  reveals  deficiencies  accom- 
ng  components  (Hammell  and  Mara, 
iciencies"  has  been  used  in  two  ways: 
ving  suboptima  1 ly,  such  as  the 
tive  in  their  application  of  proba- 
of  hypotheses,  and  (2)  to  refer  to 
s memory,  attention  span,  and 
of  most  people  to  weigh  more  than 
at  a preference  among  alternatives 


Different  approaches  to  correct  deficiencies  in  human  decision  making 
process  have  been  suggested  (Schrenk,  1969).  There  are  basically  three  ways 
to  improve  the  performance  of  the  human  decision  element  in  a system: 

a.  Selection:  insure  that  decisions  are  made  only  by  individuals 
who  are  competent  to  make  them. 

b.  Decision  Aiding:  provide  decision  makers  with  procedural  and 
technical  aids  to  compensate  for  their  own  limitations. 


c. 


Decision  Training:  attempt  to  improve  the  decision-related 
skills  of  people  in  decision-making  positions. 


Decision  aiding 
another.  While 
by  training  out 


and  decision  training  can  be  viewed  as  complementary  to  one 
decision  training  attempts  to  improve  decision  making  behavior 
deficiencies  and  highlighting  limitations,  decision  aiding 
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provides  the  decision  maker  with  procedural  and  technical  aids  which  let  him 
go  beyond  his  own  limitations  in  the  process  of  decision  making.  Much  of  the 
research  work  has  been  performed  on  decision  aiding  while  decision  training 
has  been  rarely  investigated.  One  objective  of  the  present  research  program 
was  to  narrow  this  gap. 

There  are  a number  of  decision  training  programs  that  are  being  pursued 
with  some  success.  These  programs  are  either  limited  to  the  abstract  decision 
making  subprocesses  without  an  integration  of  the  decision  training  with  the 
area  of  the  trainee’s  expertise,  or  they  highly  concentrate  on  the  task- 
specific  knowledge  and  fail  to  provide  the  trainees  with  a competent  gener- 
alized decision  making  behavior.  Training  programs  based  on  the  first 
approach  do  not  provide  the  required  link  between  decision  training  and  the 
application  area.  Programs  with  the  second  approach  suffer  from  their  short- 
comings in  three  areas: 

a.  Lack  of  decision  training  relevant  to  a new  domain,  in  case  of  the 
trainee's  transfer  to  other  task  domains. 

b.  Loss  of  decision  training  effectiveness  over  time,  since  the 
characteristics  of  decision  tasks  within  any  specific  domain 
are  subject  to  change  with  time. 

c.  Lack  of  carry  over  of  the  decision  making  capability  into 
incidental  functions,  which  are  usually  left  untrained. 

To  avoid  the  shortcomings  of  the  existing  programs,  a decision  training 
methodology  must  be  developed  that  allows  incorporation  of  general  purpose 
decision  training  components  into  task  specific  decision  training  programs. 
Since  many  problems  encountered  in  the  development  of  such  a decision 
training  methodology  have  not  been  previously  addressed,  and  since  the 
development  process  involves  a cross-section  of  different  disciplines,  such 
a development  provided  a desirable  area  for  research  and  development. 

The  research  and  development  process  provided  for  incorporation  of 
different  areas  of  knowledge  into  the  development  of  an  universal  decision 
training  methodology.  These  areas  of  knowledge  include  decision  analysis, 
psychology,  training,  and  computer  science.  Different  components  of  a 
decision  process  were  analyzed  and  the  required  behaviors  involved  in  these 
components  were  identified.  The  cognitive  complexity  of  each  identified 
behavior  was  recognized  and  the  training  behavioral  objectives  were  defined. 
The  best  instructional  method,  media,  and  content  for  accomplishing  the 
training  objectives  were  identified. 

CURRENT  DECISION  TRAINING  PROGRAMS.  There  are  a number  of  decision  training 
programs  that  are  being  pursued  with  some  success.  These  programs  are  either 
task  specific  or  treat  only  limited  aspects  of  the  decision  making  process. 
Some  of  the  programs  are  used  in  operational  contexts. 

Einhorn  and  Hogard  (1975)  developed  an  approach  to  teaching  multi- 
attribute utility  analysis  to  top-level  executives  and  middle-level  managers. 
While  their  system,  termed  "An  Idiot's  Guide  to  Decision  Making"  (Goodman 
et  al,  1976),  maintains  a reasonable  degree  of  independence  with  respect  to 
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any  specific  domain,  it  covers  only  one  method  of  alternative  evaluation. 

Other  subprocesses  involved  in  the  decision  making  process,  as  well  as  the 
interrelationships  among  the  subprocesses  in  the  procedural  body  of  the 
decision  making  system,  are  ignored. 

Decisions  and  Designs,  Inc.  (DDI)  has  developed  a Rapid  Screening 
Decision  Training  System.  The  system  involves  an  interactive  computer  pro- 
gram that  simplifies  a decision  analysis  by  focusing  on  a limited  number  of 
alternatives  and  the  major  causes  of  uncertainty  (Selvidge,  1976).  The 
training  aspect  of  the  system  consists  of  displaying  the  expected  value 
associated  with  each  alternative  evaluation  and  does  not  cover  other  elements 
of  decision  making  such  as  problem  recognition,  alternative  development,  and 
the  optimal  sequencing  and  effort  allocation  for  these  subprocesses. 

Hammond,  et  al  (1975)  present  judgment  as  the  key  elements  of  a decision 
making  process.  They  assume  that  if  people  are  taught  the  theory  behind 
judgment  analysis  and  then  trained  in  increasingly  difficult  applications  of 
task  situations,  they  will  eventually  be  able  to  analyze  any  problem  properly. 
Based  on  this  theory,  their  training  system  focuses  strongly  on  judgmental 
aspects  and  ignores  the  other  elements  of  the  decision  making  process. 

The  Decision  Analysis  Group  at  Stanford  Research  Institute  (SRI)  con- 
ducts several  different  kinds  of  training  programs  for  decision  makers 
(Goodman,  et  al , 1976).  Through  these  programs,  trainees  are  expected  to 
learn  that  the  decision  theoretic  methodology  exists,  that  uncertainties 
and  utilities  can  be  quantitatively  estimated  and  that  they  can  begin  to 
structure  and  work  through  their  own  decision  problems.  While  the  programs 
are  enriched  by  a reasonable  degree  of  generality  and  completeness,  they 
do  not  provide  the  required  link  between  decision  training  and  the  specific 
application  area. 

Michigan  State  University's  Medical  School  approach  to  the  training  of 
physicians  is  based  on  a total  curriculum  desiqn  wherein  decision  analysis 
is  integrated  within  specific  content  areas  (All al , 1973;  Elstein,  Shulman, 
and  Sprafka,  1977).  It  assumes  that  the  diagnostic  phase  of  medicine  con- 
sists of  generating  hypotheses  about  what  the  medical  problem  might  be, 
distinguishing  the  relevant  from  the  irrelevant  features  of  the  case,  and 
then  systematically  gathering  information  to  test  and  compare  alternative 
hypotheses.  The  program  is  highly  task-specific  and  it  relies  heavily  on 
the  case-study  method. 

The  Los  Angeles  Police  Academy's  "shoot/no  shoot"  training  is  an  example 
of  a task-specific  decision  training  program.  Although  the  program  covers  no 
formal  training  in  either  probability  theory  or  decision  theory,  it  includes 
extensive  courses  in  the  established  important  attributes  that  should  be 
considered  when  deciding  whether  or  not  to  shoot  in  a given  situation. 

There  is  no  relative  weighting  scheme  for  the  attributes  nor  decision  rules 
that  translate  the  utilities  of  the  attributes  into  a decision.  There  are, 
however,  general  guidelines  that  help  the  cadets  make  the  decision.  Due  to 
time  criticality,  the  cadets  are  taught  to  prune  the  decision  tree  before 
the  actual  decision  situation  arises.  The  training  system  is  tailored  for 
the  specific  task  involved  and  lacks  the  generality  and  completeness  desired 
in  a decision  training  program. 
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The  review  of  current  decision  training  programs  seems  to  reveal  a 
lack  of  completeness,  generality,  and  structural  guidance  in  the  system 
under  development  or  currently  in  use.  These  three  features  are  essential 
to  a decision  training  program  with  reasonable  generality.  A research  and 
development  project  for  a decision  training  methodology  embodying  these 
three  features  was  included  in  the  objectives  of  this  program. 

REQUIREMENTS  FOR  A GENERALIZED  DECISION  TRAINING  PROGRAM.  Current  efforts 
in  the  development  of  decision  training  systems  have  usually  evolved  in 
the  context  of  task-specific  systems  (e.g.,  medical,  tactical,  managerial, 
etc.).  A training  system  for  decision  makers  that  is  developed  within  the 
context  of  such  a task-specific  domain,  without  incorporation  of  general 
purpose  decision  training  components,  does  not  provide  trainees  with  the 
skills  required  for  making  task-specific  decisions  within  other  domains. 
Furthermore,  since  the  characteristics  of  decision  tasks  within  any  specific 
domain  may  be  subject  to  change  with  time,  such  training  systems  will  not 
even  provide  useful  long-term  guidelines  within  the  trainees'  own  domain. 

In  these  systems,  the  decision  maker  learns  a tailor-made  set  of  conceptual 
components,  procedures,  techniques,  heuristic  rules,  parameters,  and  values 
unique  to  that  particular  system  at  the  time  of  training.  The  set  of 
concepts  that  the  potential  decision  maker  learns  is  in  reality  a very 
specific  sub-set  of  more  general  conceptual  components,  procedures,  and 
techniques.  Yet,  he  never  becomes  aware  that,  in  fact,  it  is  a sub-set 
of  the  more  general  set  because  it  is  outside  the  training  boundary.  Hence, 
if  a decision  maker  is  transferred  to  a new  decision  making  domain,  or  the 
characteristics  of  the  decision  tasks  within  his  decision  domain  change,  he 
typically  has  two  choices:  he  must  either  enter  a new  task-specif ic  decision 
training  program  (and  learn  another  unique  conceptual  sub-set  again  without 
relaxing  its  relationship  to  the  first  sub-set  learned),  or  he  must  learn 
to  invent  his  own  conceptual  set  without  the  benefit  of  a formal  training 
program.  In  the  latter  case,  the  decision  maker  is  without  a formal  guide 
and  often  attempts  to  "cram"  his  new  world  into  a conceptual  sub-set  learned 
for  an  entirely  different  decision  task. 

Considering  the  fact  that  the  characteristics  of  decision  tasks  within 
any  specific  domain  are  subject  to  change  with  time,  even  decision  training 
systems  designed  for  specific  domains  must  include  components  of  general 
purpose  decision  training.  A comparative  analysis  of  general  purpose  versus 
task  specific  decision  training  will  identify  the  required  components  of 
domain  independent  decision  training  that  must  be  included  in  a task  specific 
decision  training  system.  Such  a comparative  analysis  was  included  in  the 
proposed  program. 

BACKGROUND 

PROBLEM  STATEMENT.  Instructional  System  Development  (ISD)  currently  provides 
a systematic  and  scientific  methodology  for  (1)  identifying  the  tasks  that 
operators  must  perform,  (2)  specifying  the  hierarchy  of  training  objectives, 
and  (3)  developing  the  procedures  and  schedules  for  training  the  operators 
to  perform  the  identified  tasks.  However,  decision  making  tasks  have 
typically  not  been  among  those  identified  and  treated  by  the  ISD  methodology. 
Decision  tasks  are  thus  neglected  in  present  training  courses.  In  most 
cases,  alternative  selection  has  been  considered  as  a well -structured 
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"cookbook"  response,  rather  than  as  a decision  consisting  of  specific 
sequences  of  tasks  to  be  trained.  In  those  cases  where  critical  decision 
points  have  been  recognized,  performance  is  assumed  to  come  "from  experience" 
rather  than  by  explicit  instruction.  Research  is  definitely  required  to 
develop  a methodology  for  identifying  and  isolating  decision  tasks  in  current 
system  training  functions,  as  well  as  for  development  procedures  for  train- 
ing decision  making. 

DECISION  MAKING.  Investigation  of  human  decision  making  has  focused  on  the 
process  of  choosing  among  alternatives  under  various  conditions  of  risk  and 
uncertainty.  Research  has  identified  sequences  of  tasks  included  in  the 
literature  on  decision  making,  the  list  given  by  Nickerson  and  Feehrer  (1975) 
provides  a comprehensi ve  set  for  purposes  of  discussion  and  comparison.  It 
consists  of  (1)  information  gathering,  (2)  data  evaluation,  (3)  problem 
structuring,  (4)  hypothesis  generation,  (5)  hypothesis  evaluation,  (6) 
preference  specification,  (7)  action  selection,  and  (8)  decision  evaluation. 

Theoretical  research  on  normative  (ideal)  decision  making  has  also 
identified  sources  of  human  decision-making  deficiencies.  Such  deficiencies 
include  stereo-typed  choice  sequences,  conservative  probability  application, 
incomplete  data  acquisition,  etc.  Several  investigators  (Hammell  and  Mara, 
1970;  Kanarick,  1969;  Pesch,  Hammel , and  Ewalt,  1974)  have  suggested  that 
decision  training  programs  can  diagnose  and  give  instruction  regarding  deci- 
sion deficiencies. 

DECISION  ANALYSIS.  Technqiues  of  decision  analysis  have  been  developed  that 
can  be  used  to  identify  the  decision  points  in  a sequence  of  action  per- 
formed by  a system  operator.  These  techniques  also  provide  a systematic 
method  for  identifying  the  decision  parameters  specific  to  the  situation  at 
hand  (i.e.,  alternative,  rules,  attributes).  Thus,  the  tasks  and  associated 
inputs  of  decision  making  can  be  explicitly  defined  for  training  purposes. 

The  multi-attribute  utility  approach  to  analyzing  decision  situations 
in  one  example  of  these  techniques  (Gardiner  and  Edwards,  1975;  Gardiner, 
1977).  Essentially,  this  approach  first  establishes  dimensions  of  value  -- 
"attributes"  for  the  relevant  set  of  decision  outcomes.  Each  outcome  to 
be  evaluated  is  located  on  each  dimension  --  i.e.,  assigned  an  attribute 
utility  --  by  means  of  observation  of  judgment.  Utility  can  be  thought 
of  as  gain  to  the  operator;  gains  can  be  positive;  negative  gains  are 
costs.  An  overall  utility  for  the  outcomes  is  achieved  by  aggregating  the 
multiple  utilities,  most  often  by  calculating  a weighted  average  over 
attributes.  The  expected  utility  of  an  outcome  is  its  aggregated  utility 
multiplied  by  its  probability  of  occuring,  and  the  usual  decision  rule  is 
to  select  the  outcome  with  maximum  expected  utility,  although  in  some  cases 
other  rules  may  be  preferred. 

DECISION  AIDING.  Introduction  of  decision  aids  to  the  decision  environment 
provides  a framework  for  famil iarizing  the  operator  with  decision  theoreti- 
cal concepts  and  decision  training  approaches.  It  has  been  shown  that 
adaptive  decision  aids  can  be  an  effective  instructional  tool  for  behavior 
change  and  performance  improvement  (May,  Crooks,  and  Freedy,  1976).  Adap- 
tive decision  aiding  has  also  been  effectively  used  to  show  that  when 
decision  models  based  on  real  operators  are  used  for  decision  making, 
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•performance  is  superior  to  that  of  the  operator  himself,  both  in  terms  of 

performance  scores  and  in  terms  of  decision  consistency  (Dawes,  1970;  Freedy, 
Davis,  Steeb,  Samet,  and  Gardiner,  1976). 

TECHNICAL  APPROACH 

GENERAL.  Attention  has  been  initially  focused  on  a single  military  system 
able  to  provide  good  examples  of  decision  making  tasks  which  are  important 
to  system  operation  and  which  are  likely  to  be  improved  by  training  in  an 
instructional  program.  This  system  is  the  Navy  SH-2F  (LAMPS)  ASW  heli- 
copter. An  instructional  course  for  the  LAMPS  crew  members  (Pilot/ATO 
and  Sensor  Operator)  has  recently  been  developed  by  Courseware,  Inc.,  San 
Diego.  In  effect,  Perceptronics  acted  as  "subject  matter  expert"  in  the 
area  of  decision  making.  The  final  output  of  the  decision  analysis,  as 
presented  in  this  report,  is  in  a form  suitable  for  incorporation  into  the 
specification  of  instructional  objectives,  the  planning  of  strategies,  the 
selection  of  media,  and  other  other  steps  required  for  the  production  of 
instruction. 

The  main  tasks  involved  in  the  current  program  effort  are  the 
following : 

a.  Analysis  of  LAMPS  Operators'  Functions 

b.  Decision  Selection 

c.  Description  of  Decision  Tasks 

d.  Recommendation  of  Decision  Aids 

These  tasks  are  described  separately  in  the  following  sections. 

ANALYSIS  OF  LAMPS  OPERATORS'  FUNCTIONS.  An  analysis  of  LAMPS  helicopter 
operators  identified  decision  points  that  occur  during  execution  of  the 
operator's  functions.  This  task  corresponds  to  the  Job  Analysis  phase  of 
the  instructional  development  process.  Decision  points  in  an  operator's 
sequence  of  functions  have  the  following  characteristics: 

a.  The  operator  is  required  to  make  a choice  between  a number 
of  alternatives. 

b.  The  criteria  for  selecting  among  the  alternatives  are  not 
completely  specified,  and  may  have  significant  judgmental 
components. 

c.  The  operator  is  required  to  either  define  the  possible  choices 
(problem  structuring)  or  evaluate  the  potential  outcomes 
(action  selection). 

Indicators  for  identifying  decision  points  in  the  operator's  functions 
include  the  following: 

♦ 

* a.  Additional  information  may  be  necessary  before  a choice  can 

be  completely  defined. 
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b.  The  possible  outcomes  of  the  choice  are  uncertain  and  some 
estimate  of  their  likelihood  may  be  required. 

c.  The  consequences  of  the  actions  are  composed  of  a number  of 
factors  that  must  be  aggregated. 

d.  The  alternative  consequences  of  the  current  action  will 
differentially  affect  future  actions. 

DECISION  SELECTION.  From  the  list  of  decisions  identified  in  the  job 
analysis,  a limited  number  were  selected  for  the  examination  of  decision 
training  methodology.  Selection  focused  on  highly  critical  decision.  The 
criteria  for  selection  were  the  following: 

a.  Safety  Criticality 

b.  Time  Criticality 

c.  Frequency  of  Occurrence 

d.  Current  Decision  Making  Effectiveness 

e.  Tractability 

f.  Demonstrability 

Candidate  lists  of  potential  decision  tasks  in  the  LAMPS  emergency  procedures 
were  ranked  by  LAMPS  expert  instructors  according  to  the  above  criteria.  A 
composite  ranking  was  then  obtained  and  provided  the  basis  for  final  decision 
task  selection.  The  results  of  this  selection  are  presented  in  Chapter  2. 

The  decisions  were  thus  chosen  to  represent  both  general  classes  of  decisions 
and  decisions  that  are  specific  to  particular  crew  positions. 

DESCRIPTION  OF  DECISION  TASKS.  This  corresponds  to  the  Task  Analysis  phase 
of  Instructional  Systems  Development.  Researchers  in  decision  making  have 
suggested  various  classifications  of  tasks  that  decision  makers  perform 
(Adelson,  1961;  Drucker,  1967;  Edwards,  1965;  Hill  and  Martin  1971;  Howard, 
1968;  and  Schrenk,  1969).  These  classifications  provide  a framework  for 
enumerating  the  tasks  that  the  LAMPS  operators  must  perform  in  each  of  the 
selected  decisions.  A preliminary  list  of  subprocesses  involved  in  gener- 
alized decision  making  include: 

a.  Recognize  the  decision  point 

b.  Formulate  the  alternative  courses  of  action 

c.  Establish  the  possible  outcomes  of  each  course 

d.  Estimate  the  multi-attribute  utility  ( i . e . , gains  or  losses) 
associated  with  each  outcome 

e.  Estimate  the  probability  of  achieving  the  various  outcomes 


1 


NAVTRAEQUIPCEN  77-C-0005-1 

f.  Apply  decision  rule(s)  or  criteria 

g.  Select  a best  course  of  action 

A detailed  analysis  of  the  selected  decision  tasks  were  performed  in 
the  second  phase  of  the  project.  The  set  of  decision  tasks  includes  both 
generalized  tasks  (i.e.,  decision  problem  structuring)  and  tasks  that  are 
unique  to  each  specific  decision  (i.e.,  list  of  possible  alternative 
classes).  The  results  of  this  analysis  appears  in  Chapter  3.  The  decision 
trees  developed  in  this  phase  are  included  in  the  appendices  of  this  report. 

DEVELOPMENT  OF  INSTRUCTIONAL  GUIDELINES.  The  steps  involved  in  the  actual 
production  of  training  materials  for  decision  tasks  include  the  specifica- 
tion of: 

a.  Objective  Hierarchies 

b.  Strategy  Planning 

c.  Media/Method  Selection 

d.  Course  Sequencing 

It  is  realized  that  training  crew  members  for  more  effective  decision 
making  is  no  trivial  matter.  Many  years  of  research  have  been  spent  on 
each  of  the  subprocesses  involved  in  decision  making.  Much  work  has  been 
devoted  to  the  area  of  probability  assessment  alone.  Nevertheless,  it 
has  been  realized  that  the  effort  in  analysis  of  requirements  and  method- 
ology for  decision  training  in  operational  systems  is  justified  by  the 
following  reasons: 

a.  The  vast  amount  of  theoretical  and  experimental  work  provides 
an  invaluable  guide  to  practical  application. 

b.  It  appears  that  much  of  the  benefits  to  be  derived  from  the 
decision  analytic  approach  lies  in  problem  definition  itself, 
that  is,  in  the  operator  recognizing  that  a decision  problem 
exists,  and  evaluating  it  in  terms  of  expected  costs  and 
expected  gains.  This  aspect  of  decision  making,  concepts  and 
basic  rules,  seems  highly  amenable  to  training. 

c.  By  restricting  initial  effort  to  a specific  system,  the  job  of 
defining  decision  problems  and  prescribing  procedures  for  train- 
ing is  simplified. 

d.  The  potential  payoff  is  great.  Tactical  decisions  involve 
equipment  and  procedures  of  even-increasing  cost.  Any  gain  in 
decision  effectiveness  realized  at  the  training  stage  will 
have  a large  multiplier  effect  over  the  total  system  life. 

A system  for  generating  guidelines  to  aid  development  of  instructional 
materials  for  each  decision  task  is  developed  in  this  report.  This  system 
provides  essential  information  to  be  incorporated  in  the  production  of 
Instructional  materials.  Such  Information  includes: 
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a.  Identification  of  decision  tasks  in  objective  hierarchies 

b.  Classification  of  decision  tasks  * 

c.  Identification  of  task-specific  and  basic  instructional  contents 

d.  Selection  of  the  most  effective  instructional  method 

e.  Selection  of  the  most  effective  instructional  media 

RECOMMENDATIONS  OF  DECISION  AIDS.  Identification  of  decision  points  and 
decision  tasks  suggests  decision  aids  that  could  be  incorporated  in  the 
operational  system.  Decision  aids  can  be  used  in  a variety  of  ways  to 
improve  the  logical  soundness  and  the  speed  of  decision  making.  In  general, 
aids  are  directed  toward  eliminating  deficiencies  and  limitations  in  human 
cognitive  performance.  As  an  example  of  eliminating  deficiencies,  an  aid 
can  be  used  to  remind  a decision  maker  to  seek  more  information  or  to 
help  avoid  a stereotyped  response.  As  an  example  of  overcoming  limitations, 
decisions  aids  can  help  evaluate  a larger  number  of  alternatives  than  an 
unaided  decision  maker  could  consider.  A variety  of  decision  aiding 
techniques  are  available,  ranging  from  printed  decision  trees  and  flow 
diagrams,  through  paper-and-penci 1 mathematical  procedures,  to  interactive 
computer-based  techniques.  The  computer-based  techniques  include  data- 
base retrieval  and  bookkeeping  functions,  graphical  problem  representation, 
hypothesis  generation,  simulation  of  outcome  sequences,  etc.  Operational 
decision  aids  have  implications  for  training,  since  (1)  they  would  be 
expected  to  reduce  or  change  the  elements  of  the  identified  decision  tasks, 
and  (2)  the  operators  would  also  require  training  on  the  use  of  any  software 
or  hardware  aids  introduced  to  the  operational  system. 

* 
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SECTION  II 
JOB  ANALYSIS 

GENERAL 

Since  the  objective  of  the  program  was  to  develop  a methodology  for 
the  development  of  decision  training  programs  in  specific  areas,  a task- 
specific  appraoch  for  the  research  and  development  program  was  employed. 
The  project  started  with  an  analysis  of  tasks  involved  in  the  operator's 
job  and  constituted  the  "Job  Analysis"  phase. 


Durinq  the  "Job  Analysis"  phase,  the  attention  was  focused  on  identi- 
fication and  selection  of  decision  tasks  in  the  SH-2F  (LAMPS)  ASW  operation. 
An  in-depth  study  of  the  entire  LAMPS  operation  was  conducted.  Fiqure  1 
presents  a detailed  diaqram  of  the  approach  followed  durinq  this  phase. 

The  followinq  is  a summary  of  this  approach  in  which  the  individual  items 
are  numerically  indicated  in  the  figure. 


a.  A preliminary  methodology  for  the  identification  and  classi- 
fication of  decision  tasks  was  developed.  This  methodology 
enables  the  separation  of  decision-related  LAMPS  tasks  from 
procedural  non-decision  tasks. 

b.  Two  volumes  of  LAMPS  instructional  "objective  hierarchies" 
were  analyzed  for  the  purpose  of  isolating  potential  areas 
for  decision  tasks.  These  two  volumes  taken  together  comprise 
the  document:  "Instructional  Design  and  Development  Objective 
Hierarchies  for  Pilot/ATO  and  Sensor  Courses,"  SH-2F  (LAMPS) 
Instructional  Systems  Development  Project  No.  N68221 -75-PR- 
S1201,  Data  Item  No.  A001 , prepared  by  Courseware,  Inc., 

7851  Mission  Center  Court,  Suite  220,  San  Diego,  CA  92108. 

c.  The  result  of  the  objective  hierarchy  analysis  was  a list  of 
potential  areas  for  decision  tasks. 

d.  An  in-depth  analysis  of  the  available  instructional  materials 
for  the  LAMPS  training  course  was  conducted.  This  consisted 
of  the  review  of  49  Pilot/ATO  tape  slide  courses,  34  crewmen 
tape  slide  courses,  and  187  Pilot/ATO  workbook  courses. 

e.  The  result  of  the  instructional  materials  analysis  was  a second 
list  of  potential  areas  for  decision  tasks  in  the  LAMPS 
operations. 

f.  From  the  two  preliminary  decision  area  lists,  composite 
detailed  list  of  decision  tasks  was  compiled. 

g.  To  prepare  for  final  decision  task  ranking  and  selection,  two 
experienced  LAMPS  instructors  and  two  student  LAMPS  trainees 
were  interviewed  with  prepared  questionnaires. 
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h.  The  results  of  the  questionnaires  provided: 

. a ranking  of  the  tentative  decision  tasks  in  order  of 
importance  and  criticality, 

. An  identification  of  major  decisions  in  each  potential  area, 
and 

. criteria  for  final  selection. 

i-  A final  composite  decision  task  ranking  and  selection  was  made 
based  upon  all  of  the  analysis  information. 


METHODOLOGY 

The  functions  of  both  the  Pilot/ATO  and  the  Sensor  Operator  were  the 
subjects  of  the  analysis  performed  in  the  first  phase  of  the  project.  A 
preliminary  analysis  of  the  job  situation  specified  the  required  steps  in 
the  technical  approach.  These  steps  include  the  definition  of  decision 
tasks,  classification  of  decision  tasks,  and  identification  of  decision 
task  areas. 

Performance  of  an  operator  in  a decision/non-decision  task-pool  envi- 
ronment obeys  the  scheme  presented  in  Figure  2.  As  the  input  to  this 
scheme,  a set  of  tasks  is  introduced  to  the  operator  as  part  of  the  normal 
LAMPS  operational  procedures.  The  set  contains  both  decision  and  non- 
decision tasks.  Since  processing  of  a decision  task  requires  a procedure 
considerably  different  from  the  one  for  non-decision  tasks,  the  first 
responsibility  of  the  operator  is  to  identify  the  task  as  either  a decision 
or  a non-decision.  Block  1 (Figure  2)  shows  such  a dichotomization . This 
block  acts  as  a "filter"  which  identifies  decision  tasks  and  passes  them 
to  block  3 for  classification. 

Non-decision  tasks  will  also  be  identified  in  block  1 and  passed  to 
block  2 for  processing.  It  is  assumed  that  block  2 contains  a strict  set 
of  predefined  procedures  for  processing  non-decision  tasks.  Instructional 
objectives,  outlines  and  materials  for  these  procedures  have  already  been 
developed  by  Courseware,  Inc.,  as  a part  of  the  current  LAMPS  pilot 
training  course. 

The  tasks  identified  as  decision-related  are  input  to  block  3,  where 
they  are  classified  as  Type  1,  2,  or  3.  Processing  of  a Type  1 decision 
task  requires  consideration  of  problem  structuring;  a Type  2 decision  task 
represents  action  selection.  Type  3 decisions  represent  a combination  of 
both  Type  1 and  Type  2 in  which  both  problem  structuring  and  action  selec- 
tion must  be  considered.  Blocks  1 and  3 form  part  of  the  methodology  to 
identify  and  classify  decision  tasks  in  the  job  analysis  phase.  The 
methodology  will  be  described  in  the  following  section. 

DECISION  TASK  DEFINITION  AND  CLASSIFICATION.  The  mechanism  of  decision 
task  identification  is  similar  to  that  of  a "filter"  which  passes  all, 
and  only,  decision  tasks.  The  characteristics  of  such  a filter  are  described 
by  the  definition  of  a decision  task: 


i 
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a.  The  objective  of  a decision  task  is  to  select  an  alternative 
from  a specified  set  of  alternatives. 

b.  This  selection  may  require  the  formulation  of  alternatives 
(problem  structuring) . 

c.  There  is  a lack  of  completely  specified  criteria  for  either 
alternative  formulation  or  alternative  selection. 

The  operator  actions  necessary  for  processing  a generalized  decision 
task  may  involve  considerations  such  as  establishing  possible  outcomes  and 
consequences  of  each  alternative,  determining  utilities  and  probabilities 
of  the  various  outcomes,  evaluating  major  attributes  of  each  available 
alternative,  and  applying  established  decision  rules  for  selecting  the 
best  course  of  action.  Some  of  these  operator  actions  are  shown  in  Figure 
3 with  their  relationships  to  problem  structuring  and  alternative  selection. 
In  general,  the  actions  of  alternative  formulation  and  outcome  formulation 
are  related  to  problem  structuring  while  actions  such  as  utility  and  proba- 
bility assessment  are  related  to  the  action  selection  process. 

Since  there  is  a considerable  distinction  between  the  two  tasks  of 
problem  structuring  and  alternative  selection,  there  is  a plausible  class- 
ification scheme  for  decision  tasks.  Such  a scheme  is  created  by  repre- 
senting a boundary  between  the  decision  tasks  requiring  problem  structuring 
and  the  ones  requiring  alternative  selection.  The  boundary  is  defined 
by  the  types  of  operator  actions  necessary  to  process  the  decision  task. 

It  is  this  distinction  which  is  used  to  identify  and  classify  specific  tasks 
in  the  LAMPS  operational  procedures. 

DECISION  TASK  SELECTION.  Once  the  decision  tasks  were  identified,  a com- 
posite ranking  method  was  used  to  select  the  final  LAMPS  decision  task 
areas  to  be  used  for  in-depth  study.  Preliminary  lists  of  potential 
decision  task  areas  were  initially  ranked  by  LAMPS  expert  instructors 
according  to  the  following  criteria: 


a. 

safety  criticality 

b. 

time  cri tical i ty 

c. 

frequency  of  occurrence 

d. 

current  decision  making  effectiveness 

Perceptronics ‘ analysts  then  ranked  the  same  preliminary  list 
the  following  criteria: 

according  to 

a. 

tractabi 1 i ty 

b. 

demonstrabil  ity 

A composite  ranking  was  then  obtained  and  served  as  the  basis 
decision  task  selection. 

for  final 

i ■ 

k 
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Alternative 


Figure  3.  Decision  Task  Components 
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INFORMATION  SOURCES 

The  main  information  sources  used  in  the  job  analysis  phase  originated 
from  three  main  categories: 

a-  Objectives  Hierarchies 

b.  Instructional  Materials 

c.  LAMPS  Instructor  Interviews 

Objective  hierarchies  for  Pilot/ATO  as  well  as  Sensor  Operator  courses 
were  used  as  the  starting  point.  They  consist  of  a complete  organization  of 
all  LAMPS  helicopter  operations  and  are  used  as  a basis  for  creating  train- 
ing courses.  The  result  gained  from  the  analysis  of  objectives  hierarchies 
led  to  promising  LAMPS  decision  task  areas.  Among  instructional  materials, 
Pilot/ATO  workbooks  were  the  subjects  of  detailed  studies.  The  tape  slides 
and  workbooks  deal  with  specific  training  subjects  and  are  presented  in  an 
informal  manner  conducive  to  efficient  training.  LAMPS  instructor  inter- 
views provided  the  required  information  not  existing  in  the  previous  two 
sources.  Potentials,  limitations,  and  efficiency  of  training  elements  of 
the  present  instructional  system  were  part  of  the  information  data  captured 
through  these  interviews.  Besides  the  three  major  sources  of  information, 
informal  interviews  of  LAMPS  student  trainees  and  inspection  of  actual 
LAMPS  helicopters  provided  additional  data  for  the  job  analysis.  Results 
from  each  of  these  information  sources  is  presented  in  the  succeeding 
sections.  A detailed  listing  of  decision  tasks  appears  in  the  appendices. 

OBJECTIVES  HIERARCHIES 

Instructional  design  and  development  of  SH-2F  LAMPS  operation  has  been 
represented  thoroughly  in  Pilot/ATO  and  Sensor  Operator  Objectives  Hierar- 
chies. The  objectives  are  structured  in  expanding  levels  of  detail  with 
each  objective  decomposed  into  component  sub-objectives.  Development  of 
instructional  materials  for  courses  as  well  as  course  sequence  are  based  on 
these  hierarchies.  However,  every  element  of  the  most  detailed  level  of  the 
hierarchical  does  not  necessarily  correspond  to  a single  course.  A course 
of  instruction  may  cover  many  training  objectives  at  different  hiearachical 
levels.  An  example  of  objectives  hierarchies  is  shown  in  Figure  4.  In  some 
cases,  as  many  as  11  levels  of  objectives  are  given.  The  sample  shown  in 
the  figure  is  only  the  top-most  level. 

The  objectives  hierarchies  were  a primary  subject  of  analysis.  This 
methodology  for  decision  task  identification  was  used  to  identify  those  ob- 
jectives which  could  be  classified  as  potential  decision  tasks.  A complete 
list  of  results  of  this  analysis  of  Pilot/ATO  and  Sensor  Operator  appear  in 
tables  in  Appendices  A and  B.  In  these  tables,  every  identified  decision 
task  is  classified  as  Type  1,  Type  2,  or  Type  3.  The  tasks  are  organized 
into  related  groups  in  which  a problem/sub-problem  relationship  exists 
between  consecutive  pairs  of  elements.  The  first  element  of  each  group 
represents  the  decision  task  captured  at  a higher  level.  Only  the  task 
area  for  the  first  element  of  every  group  is  identified  since  the  other 
elements  of  the  group  consist  of  the  subproblems  of  the  first  element. 
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1.0 

1.1 

1.2 

1.3 

1.4 

1.5 


1.6 


1.7 

1.8 


Perform  all  Pilot/ATO  functions  flying  an  SH-2F. 

Fly  an  SH-2F  to  complete  an  ASW  mission. 

Fly  an  SH-2F  to  complete  an  ASMD  mission. 

Fly  an  SH-2F  to  complete  a surveillance  mission. 

Fly  an  SH-2F  to  complete  a search  and  rescue  mission. 

Fly  an  SH-2F  to  complete  an  external  cargo  mission. 

Fly  an  SH-2F  to  complete  an  Internal  cargo/passenger 
transfer  mission. 

Fly  an  SH-2F  to  complete  a Naval  gunfire  spotting 
mission. 

Oescribe  the  main  components  and  operation  of  SH-2F 
aircraft  systems. 


Figure  4.  Objectives  Hierarchies 
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The  identification  and  classification  process  for  decision  task 
followed  the  methodology  explained  earlier.  Due  to  the  large  number  of 
existing  tasks,  the  plausibility  of  computerization  of  such  an  analysis 
was  considered.  A correspondence  between  the  decision-related  tasks  in 
the  objectives  hierarchies  and  certain  "keywords"  used  in  the  description 
was  noted  during  the  analysis.  A keyword,  in  this  case,  refers  to  an  element 
of  a set  of  verbs  appearing  in  an  imperative  sentence,  for  example,  "Deter- 
mine whether  to  land  the  aircraft  or  to  abort  the  recovery"  (p.  112,  Pilot/ 
ATO  Objective  Hierarchies)  or  "Perform  a search  in  accordance  with  NWP-37 
and  NATOPS"  (p.  130  Pilot/ATO  Objective  Hierarchies ) . The  description 
of  every  task  appearing  in  objectives  hierarchies  includes  at  least  one 
of  these  keywords. 

The  result  of  examination  of  objectives  hierarchies,  however,  revealed 
an  inconsistency  in  the  use  of  keywords.  Some  critical  keywords  appeared 
in  both  decision  tasks  as  well  as  non-decision  tasks.  For  example,  "state" 
and  "calculate"  appeared  mainly  in  non-decision  tasks;  however,  about  20% 
of  their  applications  were  in  definitions  of  decision  tasks.  Future  con- 
sistency in  the  use  of  keywords  for  decision  and  non-decision  tasks  is 
strongly  recommended.  A list  of  the  two  classes  of  keywords  appears  in 
Table  1.  In  spite  of  the  inconsistency  in  use,  the  keywords  provided  a 
reasonable  starting  point  for  identification  of  decision  tasks.  The  use 
of  a computer  program  for  this  part  of  decision  task  identification  could 
be  a means  of  increasing  the  efficiency  of  identification  process. 

In  summary,  the  analysis  of  the  objectives  hierarchies  led  to  the 
conclusion  that  most  of  the  decision  tasks  would  be  found  in  the  emergency 
and  tactical  operational  procedures.  Furthermore,  the  Pilot/ATO  tasks 
contained  many  more  decision-related  situations  than  the  sensor  operator 
tasks . 

INSTRUCTIONAL  MATERIALS 

Analysis  of  objectives  hierarchies  identified  the  major  areas  for 
decision  making  tasks.  In  this  section  of  the  job  analysis,  the  study  was 
focused  on  the  instructional  materials.  Two  out  of  four  existing  types  of 
course  materials  have  the  major  role  in  the  instruction  of  LAMPS  operation. 

a.  Tape  Slides  (TS) 

b.  Work  Book  (WB) 

c.  Computer  Aided  Instruction  (CAI) 

(not  included  in  the  analysis  due  to  unavailability) 

d.  Visual  Tapes  (VT) 

(not  included  in  the  analysis  due  to  unavailability) 

TAPE  SLIDES.  A series  of  audio  cassette  tapes  together  with  corresponding 
synchronized  slides  are  used  to  cover  part  of  the  course  instruction  in 
LAMPS  operation.  Tape  slides  for  Pilot/ATO,  as  well  as  crewnen,  were  the 
subject  of  a detailed  analysis.  Tape  slide  materials  mainly  represent 
technical  and  introductory  courses.  The  majority  of  tape  slide  courses 
cover  the  procedural  tasks  with  no  decision  making  steps.  These  were 
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TABLE  1.  OBJECTIVES  HIERARCHY  KEYWORDS 


DECISION  MAKING  KEYWORDS 


DETERMINE 

INTERPRET 

CALCULATE 

REVISE 

STATE 

SELECT 

PRESCRIBE 

INDICATE 

IDENTIFY 

RECOGNIZE 

CLASSIFY 

ESTIMATE 

NON-DECISION  MAKING  KEYWORDS 


CALCULATE 

PERFORM 

RECOVER 

PLAN 

CONDUCT 

DEMONSTRATE 


LIST 

NAME 

DESCRIBE 

STATE 


> 


i 
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termed  Non-Decision  Tasks  (ND).  Some  of  the  courses  represented  by  tape 
slides  covered  decision  situations  in  which  the  pilot  has  the  full  respon- 
sibility of  decision  making  (Decision  Making  Tasks,  DM).  In  the  remainder, 
the  decision  making  tasks  were  assumed  to  be  performed  by  the  pilot  through 
the  execution  of  a predefined  procedure.  These  were  termed  Decision 
Execution  Tasks  (DE). 

In  some  cases,  the  decision  making  characteristics  of  a particular 
situation  are  explicitly  emphasized  in  the  tape  slide  presentation.  The 
trainee  is  told  that  a decision  task  exists  and  is  given  situations  to 
analyze.  However,  it  is  made  exceptionally  clear  the  the  final  choice  of 
action  rests  with  him  (the  pilot  or  the  crewman)  and  that  the  basis  for 
the  decision  is  his  own  judgment  and  experience.  These  decision  tasks  are 
"obvious"  in  the  sense  that  they  are  termed  "decision  tasks"  in  the  in- 
structional materials  themselves.  In  other  cases,  a decision  task  is 
evident  in  the  instructional  materials  but  is  not  presented  to  the  student 
as  such.  These  decision  tasks  are  "hidden"  within  established  procedures 
for  executing  a particular  operational  plan. 

The  results  of  tape  slide  analysis  for  49  Pilot/ATO  and  34  Crewmen 
courses  appear  in  Appendices  C and  D.  Task  number,  type,  and  keyword  columns 
are  identical  to  the  ones  of  objectives  hierarchies  analysis.  The  task 
name  identifies  the  course  title  and  the  slide  number  represents  the  new 
codes  replacing  the  previous  coding  system  to  identify  different  tape 
slides.  The  non-avai Table  tape  slides  are  marked  by  NA.  Some  of  the  tasks 
are  described  in  the  notes  following  the  result  of  the  analysis.  These 
tasks  are  marked  by  an  asterisk  on  their  slide  numbers. 

WORKBOOKS.  A considerable  portion  of  the  LAMPS  operational  instructions  are 
covered  by  workbooks.  Figure  5(a)  through  6(b)  shows  examples  of  work- 
book courses  for  a "Radar  Offset"  approach  to  an  unidentified  contact  and 
procedures  for  an  "Immediate  Ditching"  of  the  helicopter,  respectively. 

In  addition  to  the  exercise  and  lesson  number,  the  first  page  of  a work- 
book course  always  contains  the  objective,  a general  list  of  actions,  and 
an  introduction  describing  the  course  material.  The  remainder  of  the 
lesson  is  usually  divided  into  procedures,  actions,  and  remarks  or  comments. 

In  Figure  5(b),  the  Radar  Offset  Approach  lesson,  the  action  performed 
for  procedure  2 is  "The  ATO  will  decide  if  a direct  or  offset  approach  will 
be  used."  Guidelines  for  this  decision  are  then  given.  This  is  an  example 
of  a Type  2 "obvious"  decision  in  which  the  ATO  must  choose  from  among  a 
specified  set  of  two  alternatives.  However,  Figure  6(b),  the  Immediate 
Ditching  lesson,  under  Section  4.B.2  shows  a hidden  decision  in  which  the 
pilot  is  not  given  guidelines  or  told  how  to  choose  an  alternative. 

The  analysis  of  the  Pilot/ATO  workbooks  included  187  lessons.  The 
results  of  this  analysis  appear  in  Appendix  E.  However,  a summary  of 
results  appear  in  Table  2. 

EXPERT  INTERVIEW 

The  information  acquisition  for  the  job  analysis  phase  was  completed 
by  interviewing  the  LAMPS  operational  experts.  Two  LAMPS  operation  instruc- 
tors as  well  as  two  LAMPS  operation  trainees  participated  in  the  interviews. 
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OBJECTIVE 


State  the  procedures  for 
performing  a radar 
approach  to  a contact. 


GENERALITY 

The  correct  procedure  for  performing  a radar 

approach  to  a contact  is:  

Find  radar  target. 

Determine  if  a direct  or  offset  approach  will  be  used 
Compute  estimated  time  on  top. 

Turn  to  an  inbound  course. 

Radar  as  desired. 

Mark  on  top  contact. 


INTRODUCTION 

When  flying  a radar  approach  to  a contact  the  senso  will  provide 
headings  and  distances  to  fly.  You  may  deceive  the  contact  into 
thinking  that  he  has  not  been  detected  by  flying  an  offset  approach. 
(An  offset  approach  entails  flying  parallel  or  perpendicular  to  the 
contact's  course  and  then  "cutting  in"  directly  to  him.)  If  you 
believe  the  contact  will  submerge  before  you  reach  hirn,  a straight 
in  approach  is  recommended  (It  is  easier  to  calculate  the  estimated 
time  on  top  flying  a straight  in  approach.) 
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RADAR  OFFSET 


PROCEDURE 


FINO  RADAR 
TARGET 


DETERMINE  IF 
A DIRECT  OR 
OFFSET  APPROACH 
WILL  RE  USED 


ACTION  PERFORMED 


Tht  sanso  ihould  locate  the  contact 
end  paia  the  bearing  and  range 
Information  to  the  cockpit. 


The  ATO  will  decide  if  e direct  or 


a.  A direct  approach  is  flown  directly 
from  the  aircraft's  present  position 
to  the  surface  contact. 


! IMPORTANT  NOTES 

I AND  COMMENTS 


a.  This  approach  is  recommended  If 
you  believe  you  have  been  detected. 

STRAIGHT  IN  -rC0NTACT 


b.  The  offset  approach  is  flown  to  a 
position  abeam  the  target  to  one 
tide  at  the  closest  point  of 
approach.  From  this  point  a 
direct  leg  is  used. 


AIRCRAFT* 

b.  If  you  think  you  are  detected  and 
Wish  to  get  closer  to  the  contact 
prior  to  starting  your  direct  run, 
you  should  use  the  offset  approach. 
If  the  contact  disappears  from  the 
radar  during  an  offset  approach, 
immediately  perform  a direct 
approach  to  the  contact. 
CLOSEST  POINT  OF  APPROACH 


CONTACT 


AIRCRAFT 


Hi 


COMPUTE  The 

estimateo 

TIME  ON  TO? 


Determine  the  d. stance  from  the 
aircraft  to  the  contact  and  adj  jst 
for  grourdipeed 


Time  on  top  is  important  in  case  your 
contact  disappears  from  the  radar 


Figure  5(b) 
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IMMEDIATE  DITCHING 


EXERCISE:  f-7 


LESSON:  Z 
SEGMENT:  7 


OBJECTIVE 


State  the  steps  to  complete  an 
immediate  ditching. 


MEDIA:  Workbook 


II 

| 

I 


GENERALITY 


1.  Alert  crew  and  passengers, 
harness  locked. 

2.  Pilot's  door  - open 

3.  Transmit  distress  message 

4.  After  water  landing: 

A.  Two  engine  fail 

1.  Full  up  collective 

2.  Rotor  brake  • on 

3.  Abandon  aircraft 

B.  Single  Eng  fail 

1.  Hold  helicopter  level 
✓ 2.  Either  secure  eng  and  rotors 
freleaslTTilrnen  a ncT~a5 ah  jjon 
aimafi_-.or-Lfpll.Qw  .jingle . eng 


COPILOT 


1.  Open  copilot's  door,  harness 
locked. 

2.  Jettison  all  external  cargo/ 
stores  and  sonobuoys. 

3.  When  rotor  stops,  abandon 
aircraft. 


INTRODUCTION 

In  the  event  of  an  immediate  ditching  situation  it  is  critical  that  all 
of  the  people  in  the  aircraft  react  to  it  and  perform  their  NAT  OPS 
procedures.  You  will  be  required  to  know  both  the  pilot  and  copilot 
duties  since  you  will  be  serving  in  both  functions  during  your  training. 
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./ 


IMMEDIATE  DITCHING  (PILOT) 


PROCEDURE 


ACTION  PERFORMED 


ALERT  CREW  AND 
PASSENGERS. 
HARNESS  LOCKED 


1 

■ 

PILOT'S  DOOR  - 
OPEN 

mm 

TRANSMIT  DIS 

TRESS 

MESSAGE 


AFTER  WATER 

LANDING: 

A.  TWO  ENGINE 
FAIL 

1.  COLLECTIVE: 
FULL  UP 

2.  ROTOR  BRAKE 
ON 

3.  ABANDON  AIR 
CRAFT 

B SINGLE  ENG 
FAIL 

1.  MOLD  HELI- 
COPTER LEVEL 

2 EITHER  SECURE 
ENG  AND 
ROTORS  AND 
RELEASE 
HARNESS  AND 
ABANDON 
AIRCRAFT  OR 
FOLLOW  SINGLE^ 
ENG  WATER 
TAKEOFF 
PROCEDURES 


Let  the  crew  know  you  are  going 
to  ditch  over  the  ICS. 


Switch  to  guard  and  tell  them 

1)  I D. 

2)  Position 

3)  Number  of  people  on  board 

4)  Problem 


mivu  Luunviuvi  o 


IMPORTANT  NOTES 
AND  COMMENTS 


If  the  Senso  is  on  private,  monit- 
oring sonobuoys.  you  will  have  to 
use  the  call  position  of  the  ICS  to 
talk  to  him. 


If  time  permits,  repeat  your  distress 
call. 


1)  Do  not  inflate  floatation  gear 
until  clear  of  aircraft 

2)  Unplug  helmet  cords  before 
trying  to  get  out  of  the  aircraft 

3)  WARNING-Oo  not  abandon 
aircraft  before  the  rotor  has 
stopped  turning. 


Figure  6(b) 
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TABLE  2.  RESULTS  OF  INSTRUCTIONAL  MATERIAL  ANALYSIS 


t 


OBVIOUS  DECISIONS 

. External  Engine  Fire  (TS) 

. Gear  Box  Oil  Malfunction  (TS) 

. Determining  Restart  Feasibility  (TS) 
. Radar  Offset  Approach  (WB) 


HIDDEN  DECISIONS 

. Combining  Gear  Box  Oil  Malfunction  (TS) 
. Hydraulic  System  Malfunction  (TS) 

. Electrically-Caused  Overspeed  (TS) 

. Loss  of  Tail  Rotor  Thrust  (TS) 

. Loss  of  Tail  Rotor  Control  (TS) 
Immediate  Ditching  (WB) 
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A series  of  predesigned  questionnaires  was  prepared  from  the  specific  tasks 
identified  in  objectives  hierarchies  and  tape  slide  analysis.  A sample 
questionnaire  appears  in  Table  3.  The  questionnaire  concerns  the  problem 
of  a gearbox  malfunction.  It  is  classed  as  an  emergency  operation.  Each 
question  attempts  to  elicit  information  about  the  particular  decision. 

Question  Q1  asks  for  a set  of  alternative  actions  which  can  be  taken 
when  a gearbox  malfunction  occurs.  Q2  tries  to  locate  attributes  of  each 
alternative  which  may  be  used  in  selection.  Since  limiting  the  initial 
set  of  alternatives  can  be  a help  in  rapid  decision  making,  question  Q3 
probes  for  situations  which  allow  the  deletion  of  one  or  more  alternatives. 
Questions  Q4  and  Q5  request  value  judgments  on  relative  utility  and  proba- 
bility of  occurrence,  respectively.  Only  a linguistic  value  judgment  is 
required  in  this  case.  Normally,  however,  numerical  judgments  are  preferred. 
Q6  is  a structural  question.  It  determines  if  this  is  a "one  shot" 
decision  situation  or  if  it  is  repetitive  in  nature.  Questions  Q7,  Q8,  and 
Q9  concern  the  "cues  and  alerts"  for  a gearbox  malfunction.  These  are 
indicators  that  a malfunction  is  either  about  to  happen  or  has  already 
occurred.  Question  Q8  deals  with  completeness.  That  is,  if  there  is  an 
inminent  gearbox  malfunction,  will  all  cues  and  alerts  be  observable? 
Question  Q9,  on  the  other  hand,  deals  with  uniqueness.  That  is,  are  there 
any  other  malfunctions  with  exactly  the  same  cues  and  alerts  as  a gearbox 
malfunction?  If  this  situation  could  occur,  there  would  exist  an  ambiguity 
in  the  recognition  of  an  emergency  decision  situation.  In  such  cases,  it 
would  be  beneficial  to  design  a new  alert  (such  as  an  emergency  indicator 
light)  to  differentiate  between  the  two  ambiguous  conditions. 

Questions  Q10,  Qll,  and  Q12  produced  the  most  valuable  information 
for  the  selection  of  decision  task  areas  for  further  study.  Q10  required 
the  participants  to  rank  the  gearbox  malfunction  with  respect  to  other 
potentially  critical  areas.  The  only  criteria  given  was  "importance" 
and  it  was  left  to  the  participant  to  define  this  word  in  his  own  terms. 
Immediately  after  this  ranking,  the  participant  was  asked  to  describe  the 
major  factors  that  he  considered  when  thinking  about  "importance."  Both 
instructors  and  both  students  gave  the  same  list  of  criteria: 

a.  Personnel  Safety 

b.  Mission  Effectiveness 

c.  Equipment  Salvage 

This  unanimous  consensus  is,  of  course,  attributable  to  the  fact  that  all 
participants  were  part  of  the  same  training  program.  However,  all  agreed 
that  not  only  were  these  the  most  important,  but  also  that  there  were  no 
other  major  ones. 

The  last  question,  Q12,  required  the  ranking  of  the  potential  decision 
areas  in  terms  of  each  criteria  separately.  The  detailed  results  of  these 
rankings  will  be  described  in  the  next  sections. 


35 


NAVTRAEQUIPCEN  77-C-0005-1 


TABLE  3. 


SAMPLE  OF  EXPERT  INTERVIEW  QUESTIONNAIRE 


) 


(Ql)  WHAT  MAY  BE  DONE  WHEN  A GEARBOX  MALFUNCTION  HAPPENS? 

(Q2 ) WHAT  ARE  THE  ADVANTAGES  AND  DISADVANTAGES  INVOLVED  IN 
EACH  ALTERNATIVE? 

(Q3)  IS  THERE  ANY  SITUATION  IN  WHICH  SOME  OF  THE  ALTERNATIVES 
ARE  USELESS?  WHICH  ONE  AND  IN  WHAT  SITUATION? 

(Q4)  ARE  ANY  OF  THE  ALTERNATIVES  MORE  PREFERABLE  IN  CERTAIN 
SITUATIONS?  WHICH  ONE  AND  IN  WHAT  SITUATION? 

(Q5)  WHAT  IS  THE  CHANCE  OF  SUCCESS  FOR  EACH  ALTERNATIVE? 
(GOOD,  BAD,  NOT  LIKELY,  ...) 

(Q6)  IF  THE  ATTEMPT  TO  PERFORM  AN  ALTERNATIVE  FAILS,  WILL 

THERE  BE  A CHANCE  TO  TRY  ANOTHER?  HOW? 

\ 

(Q7)  WHAT  ARE  THE  CUES  AND  ALERTS  FOR  SUCH  A MALFUNCTION? 

(Q8)  DO  ALL  CUES  EXIST  IN  CASE  OF  THIS  MALFUNCTION? 

(Q9)  IS  THERE  ANY  OTHER  MALFUNCTION  WITH  THE  SAME  CUES? 

(QIO)  HOW  DO  YOU  RANK  THE  IMPORTANCE  OF  SUCH  A MALFUNCTION? 
(Qll)  WHAT  WAS  THE  CRITERIA  FOR  IMPORTANCE  FOR  THIS  RANKING? 
(Q12)  RANK  THE  TASK  FOR  EACH  CRITERION  SEPARATELY. 


36 


NAVTRAEQUIPCEN  77-C-0005-1 


CANDIDATE  DECISION  TASKS 

Analysis  of  instructional  materials  identified  the  decision  task  areas. 
This  information,  together  with  the  potential  decision  task  areas  detected 
in  objectives  hierarchies  analyris  (and  confirmed  in  expert  interviews), 
produce  the  list  of  final  Pilot/ATO  decision  tasks.  The  list  shown  in 
Table  4 contains  fifteen  decision  task  areas  covering  a variety  of  oper- 
ational procedures.  The  list  formed  the  basis  for  the  selection  of  two 
major  areas  for  in-depth  analysis  in  the  next  phase. 

SELECTION  CRITERIA  AND  ANALYSIS 

Two  classes  of  criteria  were  identified  for  the  final  selection  of 
decision  tasks.  The  first  class  contains  the  criteria  of  importance  as 
judged  by  LAMPS/ASW  operational  experts.  This  class  includes: 

a.  Safety  criticality 

b.  Time  criticality 

c.  Frequency  of  occurrence 

d.  Current  decision  making  effectiveness 

The  second  class  concerns  criteria  for  usefulness  of  the  task  to  demon- 
strate the  methodology  of  decision  training.  The  two  criteria  of  this 
class  were: 

a.  Tractability 

b.  Demonstrability 

where  tractability  is  an  aggregation  of  three  basic  components: 

a.  Representativeness 

b.  Size 

c.  Degree  of  Abstraction 

Since  all  the  criteria  were  equally  important  to  the  object  of  the  task 
selection,  an  equal-weight  (w=l.)  utility  model  was  used  to  make  the  final 
selection.  A weight  of  w=-l  was  given  for  "current  decision  making  effec- 
tiveness" since  this  reflects  the  negative  effect  of  this  criteria  on  the 
task  selection. 

The  detailed  analysis  chart  for  the  decision  task  selection  appears  in 
Table  5.  The  entries  of  each  column  are  the  result  of  the  ranking  of  the 
candidate  decision  tasks  by  the  interview  participants  with  respect  to  the 
corresponding  criteria.  The  last  column.  Mean  Ranks,  is  an  indication  of 
priority  for  selection  of  each  task  according  to  the  above  criteria  and 
evaluation  scheme. 
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TABLE  4.  FINAL  CANDIDATE  PILOT  ATO  DECISION  TASKS 


FLYING  METHOD  (OH) 

MISSION  SAFETY  (II) 

TYPE  OF  LANDING  (II) 

LAND  OR  WAVE  OFF  (II) 

ABORT  OR  CONTINUE  THE  MISSION  (OH) 
DITCHING  SITUATION  (WB) 

RESTART  FEASIBILITY 
RADAR  APPROACH  (WB) 


ENGINE  QUITS  (OH) 

EQUIPMENT  MALFUNCTION  (OH) 

GEARBOX  MALFUNCTION  (TS) 

HYDRAULIC  SYSTEM  MALFUNCTION  (TS) 
LOSS  OF  ROTOR  THRUST  (TS) 

LOSS  OF  TAIL  CONTROL  (TS) 

FIRE  (TS) 


OH:  OBJECTIVE  HIERARCHY 
II:  INSTRUCTOR  INTERVIEW 
WB:  WORK  BOOK 
TS:  TAPE  SLIDE 
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TABLE  5.  ANALYSIS  OF  DECISION  TASK  SELECTION 


Decision  Task 

Safety 
f r i t if  al  i ty 
(w-l) 

T imp 

Cri tica 1 i ty 
(*  1) 

F requenc y 
of 

Occurrence 
( w-  1 ) 

Current  DM 
Effect i venpss 
<*--!) 

Tractabi  1 i ty 

(w-1) 

Demonstrabl 1 1 ty 

(»'D 

Mean 

Ranks 

Flying  Method 

M 

1? 

9 

4 

7 

8 

46 

Mission  Safety 

13 

11 

8 

5 

IS 

15 

57 

Type  of  Landing 

1? 

13 

1 

3 

6 

7 

35 

Land  or  Wavp  Off 

9 

10 

7 

6 

4 

3 

22 

Abort  or  Continue  th p Mission 

1) 

1 

4 

9 

3 

4 

35 

Ditching  Situation 

5 

8 

15 

8 

1 

1 

22 

Restart  Feasibility 

1 

9 

13 

7 

8 

5 

35 

Radar  Approach 

IS 

IS 

5 

1 

13 

6 

S3 

Engine  Quits 

1 

3 

12 

13 

2 

2 

7 

Equipment  Malfunction 

10 

14 

3 

2 

5 

9 

39 

Gearbox  Malfunction 

4 

4 

7 

12 

10 

11 

24 

Hydraulic  System  Malfunction 

8 

t 

6 

11 

11 

12 

31 

loss  of  Rotor  Thrust 

2 

1 

14 

IS 

9 

10 

21 

1 oss  of  fa  f 1 Control 

6 

6 

11 

10 

12 

14 

39 

F ire 

3 

7 

10 

14 

14 

13 

28 

(w~weiqht ) 
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The  ranks  in  each  column  were  obtained  as  follows.  Three  participants 
independently  ranked  the  tasks  under  safety  criticality.  Two  of  these 
rankings  were  almost  identical  and  one  varied  greatly.  The  ranking  which 
was  at  variance  was  disregarded,  and  the  other  two  were  averaged  and  re- 
ranked. The  three  criteria,  Time  Criticality,  Frequency  of  Occurrence, 
and  Current  Decision  Making  (DM)  Effectiveness  were  ranked  by  one  par- 
ticipant each.  The  two  criteria,  Tractability  and  Demonstrabil ity , were 
ranked  by  Perceptronics'  analysts  before  the  interviews. 

RECOMMENDED  DECISION  TASKS 

An  additive  model  with  unit  weights  was  used  to  determine  the  final 
composite  ranking.  The  numbers  in  the  Mean  Ranks  column  in  Table  5 are 
a result  of  adding  together  the  rank  values  of  each  of  the  criteria  columns 
with  the  exception  of  Current  DM  Effectiveness  which  was  subtracted.  The 
final  ranking  is  shown  in  Table  6.  "Engine  Ouits"  was  by  far  the  most 
important  with  "Loss  of  Tail  Rotor  Thrust,"  "Land  or  Wave  off,"  "Ditching 
Situation,"  and  "Gearbox  Malfunction"  closely  grouped.  In  this  group, 

Land  orWaveoff  was  chosen  since  it  would  provide  possible  experimentation 
without  danger.  Loss  of  Tail  Rotor  Thrust  and  Gearbox  Malfunction  are 
technical  tasks  similar  to  Engine  Quits.  Ditching  Situations  are  a direct 
result  of  Engine  Quits  and  could  benefit  from  this  analysis.  Table  7 
presents  characteristics  of  the  two  selected  decision  tasks  and  compares 
and  contrasts  them.  The  selection  decision  tasks  for  in-depth  analysis: 
Engine  Quits  and  Land  orWaveoff,  represent  a rich  environment  for  appli- 
cation of  decision  training  methodology. 
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TABLE  6.  SELECTION  ANALYSIS  RESULTS  AND  RECOMMENDATION 


*1. 

ENGINE  QUITS 

8. 

RESTART  FEASIBILITY 

2. 

LOSS  OF  ROTOR  THRUST 

9. 

ABORT  OR  CONTINUE  THE  MISSION 

*3. 

LAND  OR  WAVE  OFF 

10. 

TYPE  OF  LANDING 

4. 

DITCH  SITUATION 

11. 

EQUIPMENT  MALFUNCTION 

5. 

GEARBOX  MALFUNCTION 

12. 

LOSS  OF  TAIL  CONTROL 

6. 

FIRE 

13. 

FLYING  METHODS 

7. 

HYDRAULIC  SYSTEM  MALFUNCTION 

14. 

RADAR  APPROACH 

15. 

MISSION  SAFETY 
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TABLE  7. 

TASK 

1.  ENGINE  QUITS 


2.  LAND  OR  WAVE  OFF 


CHARACTERISTICS  OF  RECOMMENDED  DECISION  TASKS 


CHARACTERISTICS 

VERY  HIGH  SAFETY  CRITICAL,  VERY  HIGH  TIME 
CRITICAL,  VERY  LOW  FREQUENCY  OF  OCCURRENCE, 

VERY  LOW  CURRENT  DECISION  MAKING  EFFECTIVENESS, 
VERY  GOOD  DECISION  TASK  REPRESENTATIVE,  VERY 
LARGE  PROBLEM  SIZE,  LOW  DEGREE  OF  ABSTRACTION, 
VERY  HIGH  DEGREE  OF  DEMONSTRABILITY , TECHNICAL 

MEDIUM  SAFETY  CRITICALITY,  LOW  TIME  CRITICALITY, 
VERY  HIGH  FREQUENCY  OF  OCCURRENCE,  MEDIUM 
CURRENT  DECISION  MAKING  EFFECTIVENESS,  VERY 
GOOD  DECISION  TASK  REPRESENTATIVE,  SMALL 
PROBLEM  SIZE,  MEDIUM  DEGREE  OF  ABSTRACTION, 

VERY  HIGH  DEGREE  OF  DEMONSTRABILITY, 

OPERATIONAL 
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*- 
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SECTION  III 

DECISION  TASK  ANALYSIS 


GENERAL 

To  identify  the  characteristics  of  the  decisions  involved  in  each 
decision  situation,  a detailed  decision  task  analysis  was  performed  on  the 
decision  situations  selected  in  the  first  phase.  This  analysis  constituted 
the  second  phase  of  the  project  and  provided  detailed  task  specific  infor- 
mation. This  information  may  be  regarded  as  secondary  instructional  con- 
tent for  the  objective  decision  training  program.  Although  the  secondary 
instructional  content  is  not  vital  to  the  training  program,  its  inclusion 
may  be  desirable  in  some  cases.  The  decision  of  whether  to  include  the 
secondary  instructional  content  will  be  made  at  the  time  of  actual  in- 
structional material  development  by  the  instruction  technologist  and 
based  on  the  desired  level  of  detail  and  degree  of  task-dependency  for  the 
objective  training  program. 

During  the  second  phase  of  the  contract,  the  "Decision  Task  Analysis" 
phase,  a detailed  analysis  of  the  selected  decision  task  areas  was  per- 
formed. The  results  of  this  analysis  led  the  program  into  two  groups  of 
activities.  The  first  group  consisted  of  further  research  activities  in 
the  task  areas  in  order  to  identify  the  characteristics  of  the  decision 
tasks  involved  as  well  as  the  required  facts  and  elements  contributing 
to  the  decision  making  process.  The  second  group  involved  theoretical 
analysis  performed  to  reveal  the  underlying  concepts,  relevant  decision 
rules,  and  optimal  procedures  for  the  process.  The  first  group  provides 
task-specific  knowledge  while  the  second  provides  the  theoretical  backup 
as  well  as  domain- independent  data  required  for  the  development  of  the 
guidelines  for  instructional  materials  design. 

Figure  7 shows  the  work  flow  diagram  of  the  Decision  Task  Analysis 
Phase.  The  following  is  a summary  of  the  work  accomplished  during  this 
period.  The  individual  items  are  numerically  indicated  in  Figure  7. 
Detailed  description  of  major  activities  are  presented  in  succeeding 
sections. 


a.  A detailed  analysis  of  the  selected  decision  task  areas  was 
performed.  A complete  study  of  the  workbook  materials  as  well 
as  tape  slides  relating  to  the  task  areas  supplied  the  formal 
data,  which,  together  with  the  expert  interview  results, 
provided  the  background  knowledge  in  the  selected  tasks. 

Since  ditching  was  recognized  as  a critical  task  area  related 
to  engine  quits,  it  was  analyzed  in  this  activity  in  addition 
to  the  two  selected  areas. 

b.  The  decision  elements  involved  in  each  decision  task  area  were 
identified.  The  alternative  courses  of  action  for  each  decision 
and  the  possible  outcomes  for  each  action  were  specified. 

c.  Based  on  the  identified  decision  elements,  the  decision  trees 
for  engine  quits,  land  or  wave  off,  and  ditching  decision  task 
areas  were  constructed. 


43 


NAVTRAEQUIPCEN  77-C-0005-1 


d.  A set  of  questionnaires  and  a number  of  scenarios  were  designed 
to  be  used  for  examination,  completion,  and  possible  modification 
of  the  generated  decision  trees.  The  questionnaires  are  designed 
to  provide  the  probabilities  and  utilities  on  the  decision  trees. 
The  scenarios  are  designed  to  test  the  structure  of  the  decision 
trees  and  acquire  possible  additional  alternatives  for  decision 
tasks  through  a protocol  analysis. 

e.  The  developed  scenarios  were  used  in  a protocol  analysis  to  test 
the  structure  of  the  decision  trees  and  acquire  possible  additional 
alternatives  for  decision  tasks.  LAMPS  operation  instructors  were 
the  participants  of  the  protocol  analysis. 

f.  The  designed  questionnaires  were  used  to  provide  the  probabilities 
and  utilities  or  the  decision  trees.  This  information  was 
acquired  through  a LAMPS  operation  expert  interview. 

g.  Based  on  complete  decision  trees,  all  decision  tasks  involved  in 
the  selected  decision  task  areas  were  identified.  A complete 
decision  solution  for  each  one  was  directly  produced  from  the 
task  area  decision  trees. 

h.  Different  decision  making  methods  and  procedures  were  analyzed 
and  the  ones  suitable  for  identified  decision  tasks  were  selected. 
The  analysis  was  heavily  based  on  the  domain-dependent  knowledge 
acquired  in  the  preceding  activities  to  provide  the  task-specific 
knowledge  required  for  development  of  the  guidelines  for  instruc- 
tional material  design. 

i.  Two  different  roles  for  a human  operator  as  a decision  element 
were  identified:  an  operator  as  a decision  maker  and  an  oper- 
ator as  a decision  executor. 

j.  A visit  to  the  Los  Angeles  Police  Academy  was  made  to  review  the 
"shoot/no  shoot"  program.  The  program  is  a decision  training 
methodology  by  simulation  focusing  on  the  problem  of  when  to 
shoot  rather  than  how  to  shoot.  Although  there  is  no  formal 
training  in  either  probability  theory  or  decision  theory,  there 
are  extensive  courses  in  the  established  important  attributes 
that  should  be  considered  when  deciding  whether  or  not  to  shoot 
in  a given  situation.  The  application  of  a similar  approach 
enriched  with  a better  understanding  of  decision  making  elements 
was  studied  for  the  training  of  time-critical  decision  tasks 
(see  Appendix  I ) . 

k.  Two  different  approaches,  generalized  and  task-specific  decision 
training,  were  analyzed  and  the  validity  of  each  one  for  the 
decision  training  program  was  studied.  In  a task-specific 
decision  training  program,  the  task-domain  knowledge  is  viewed 
as  a nucleus  around  which  decision  training  will  be  built.  In 
this  case,  decision  training  development  starts  from  the  most 
task-dependent  level  and  progresses  by  addition  of  less 
task-dependent  levels.  However,  a generalized  decision  training 
program  must  be  developed  without  an  exact  knowledge  of  the 
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application  domain.  Therefore,  the  development  of  the  train- 
ing program  must  be  such  that  it  "contains"  all  possible  appli- 
cation domains.  In  this  approach,  the  program  starts  with  the 
most  general  levels  of  decision  training  and  progresses  to  the 
ones  with  a lower  degree  of  generality. 

l.  A comprehensive  list  of  words,  phases,  and  terms  were  classified 
into  two  groups  of  decision  related  and  unrelated  "behaviors." 

The  decision  related  group  provides  a list  which  should  be  used 
to  define  the  instructional  objectives  of  the  decision  program. 

m.  The  decision  making  subprocesses  were  decomposed  into  relevant 
behaviors  that  the  trainee  must  gain  by  the  end  of  the  decision 
training  program.  The  list  of  relevant  behaviors  recognized  by 
this  decomposition  will  provide  a task-independent  set  of  data 
required  to  construct  instructional  objectives  which,  in  turn, 
will  be  used  in  the  development  of  instructional  materials. 

METHODOLOGY 

A series  of  activites  in  the  second  phase  of  the  project  involved 
theoretical  analysis.  The  activities  include: 

a.  Identification  of  decision  maker/decision  executor 

b.  Identification  of  decision-related  behaviors 

c.  Analysis  of  general ized/task- specif ic  decision  training 

d.  Decomposition  of  decision  subprocesses  into  relevant  behavioral 
objectives. 

These  activities  constitute  a basis  for  development  of  the  methodology 
designed  for  the  decision  training  program.  Items  (a)  and  (b)  as  well  as 
an  overview  of  decision  tree  representation  are  described  in  this  section. 

DECISION  MAKER  VERSUS  DECISION  EXECUTOR.  Analysis  of  the  possible  roles  of 
a human  operator  as  a decision  element  provides  information  concerning  the 
nature  of  the  training  task.  Such  an  analysis  was  performed,  and  two 
different  roles  for  a human  operator  as  a decision  element  were  identified: 
an  operator  as  a decision  maker  and  an  operator  as  a decision  executor. 

The  distinction  between  the  two  roles  can  be  defined  by  the  degree  of  aiding 
that  is  provided  to  the  operator  at  the  time  of  interaction  with  the 
decision  problem.  If  the  decision  problem  has  been  analyzed  prior  to  its 
actual  occurrence  and  a general  solution  in  terms  of  a reasonably  specified 
procedure  has  been  developed,  the  responsibility  of  the  operator  involved 
with  the  decision  problem  is  to  "execute"  the  procedure  with  the  consid- 
eration of  the  problem-envi ronmental  conditions.  However,  if  such  a 
procedure  does  not  exist,  the  operator  will  also  be  responsible  for  gen- 
erating the  procedure  for  selecting  the  decision  solution  (Figure  8). 

For  critical  decision  tasks,  the  decision  making  performance  can  be 
increased  by  shifting  the  role  of  operator  from  decision  maker  to  decision 
executor  as  much  as  possible.  This  objective  may  be  partly  accomplished 
through  training. 
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CURRENT 

ENVIRONMENTAL 

CONOITION 


Figure  8.  Decision  Maker/Decision  Executor 
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DECISION  RELATED  BEHAVIORS.  The  success  of  any  decision  training  program 
can  be  evaluated  by  measuring  the  degree  of  improvements  in  a set  of 
desirable  behaviors  provided  by  the  training  program  at  its  conclusion. 

The  set  of  behaviors  includes  the  relevant  cognitive  abilities  required 
to  accomplish  the  goals  of  the  training  program:  effective  decision  mak- 
ing in  a specific  environment.  Such  a set  constitutes  the  decision 
training  behavioral  objectives.  *The  first  step  to  identify  the  decision 
training  behavioral  objectives  is  an  analysis  of  different  training  be- 
havioral terms.  Such  analysis  was  performed  and  decision-related  be- 
havioral terms  for  different  classes  of  training  were  identified  (Table  8). 

DECISION  TREE  REPRESENTATION . A decision  problem  and  its  solution  may  be 
represented  in  different  formats.  Decision  trees  constitute  a structured 
format  for  such  a purpose.  Since  decision  trees  provide  the  most  descrip- 
tive means  for  representation  of  decision  problems,  they  were  used  to 
describe  the  results  of  the  decision  task  analysis  performed  for  each  of 
the  selected  decision  tasks.  This  method  of  representation  is  described 
briefly  in  this  subsection. 

A decision  tree  is  a structural  representation  of  sequences  of  antic- 
ipated decisions  and  events  in  a specific  environment.  A conventional 
decision  tree  is  composed  of  three  different  nodes:  decision  nodes, 
event  nodes,  and  terminal  nodes.  A decision  node  represents  the  alter- 
native courses  of  action  possible  at  a situation  described  by  the  path 
from  the  root  of  the  tree  to  the  decision  node  (normally  indicated  by 
a small  square  --  Figure  9a).  An  event  node  represents  different  out- 
comes which  might  occur  as  a result  of  application  of  a specific  alterna- 
tive course  of  action  in  a specific  situation  described  by  the  path  from 
the  root  of  the  tree  to  the  preceding  decision  node.  An  event  node 
is  normally  represented  by  a small  circle  (Figure  9b).  A probability 
of  occurrence  is  assigned  to  each  branch  emerging  from  an  event  of  the 
corresponding  event.  The  probabilities  of  events  associated  with  each 
event  node  must  sum  to  one.  That  is,  the  outcomes  at  each  event  node 
must  be  mutually  exclusive  and  exhaustive.  A terminal  node  represents  an 
outcome  which  can  be  judgmentally  assessed  with  reasonable  accuracy.  Such 
assessment  provides  a utility  number  which  is  a quantitative  evaluation 
of  the  degree  of  desirability  of  the  corresponding  outcome  in  the  context 
of  the  specific  situation.  In  some  cases,  due  to  the  ease  of  assessment, 
outcomes  are  assessed  in  terms  of  losses  involved  instead  of  utilities. 

In  such  cases,  a loss  number  at  each  branch  represents  the  degree  of 
aversion  for  the  corresponding  outcome  in  the  context  of  the  specific 
situation.  Terminal  nodes  are  represented  by  a small  triangle  (Figure  9c). 

Analysis  of  different  decision  tasks  revealed  the  need  for  the  intro- 
duction of  a new  type  of  node  into  decision  trees.  Such  a node,  called  a 
"situation  node,"  represents  possible  situations  which  might  be  encount- 
ered at  a specific  point  on  the  decision  tree.  At  the  planning  stage  of 
a decision  process,  the  involved  person  (the  decision  maker)  has  only  a 
knowledge  concerning  all  possible  situations  that  may  exist.  However, 
unlike  the  case  of  event  nodes,  at  the  actual  decision  implementation 
time  (immediately  before  the  decision)  the  involved  operator  (the  decision 
executor)  will  know  with  certainty  which  of  the  possible  situations 
actually  exists.  Therefore,  treatment  of  such  situations  was  an  event 
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TABLE  8.  DECISION-RELATED  BEHAVIORAL  TERMS 


General  Instructional  Objectives  General  Discriminative  Behaviors 


Analyze 

Evaluate 

Choose 

Define 

Apply 

Recognize 

Describe 

Detect 

Compute 

Think 

Differentiate 

Distinguish 

Use 

Identify 

Select 

Complex,  Logical, 

Judgmental 

Laboratory  Science 

Behaviors 

Behavior 

Analyze 

Compare 

Apply 

Cal ibrate 

Decide 

Evaluate 

Conduct 

Decrease 

Formulate 

Generate 

Increase 

Plan 

Induce 

Infer 

Weigh 

Plan 

Creative  Behavior 

Mathematical  Behavior 

Generalize 

Modify 

Compute 

Count 

Predict 

Question 

Estimate 

Measure 

Rearrange 

Verify 

Calculate 

Mi  seel laneous 

Consider  Determine 

Discover 


* 
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node,  and  assignment  of  subjective  probabilities  will  not  provide  a 
successful  representation  for  the  problem.  Situation  nodes  are  repre- 
sented by  a small  circle  surrounding  an  "X"  (Figure  9d). 

An  example  of  a situation  node  arises  when  the  "Engine  Quits:  situa- 
tion is  encountered.  A decision  maker  (may  be  the  pilot  himself),  who  is 
analyzing  this  situation  prior  to  actual  occurrence,  knows  that  such  a 
situation  may  be  caused  by  either  a single  engine  failure,  a dual  engine 
failure,  or  a fuel  starvation.  He  is  also  aware  of  the  fact  that  each 
of  these  situations  are  different  in  nature  and,  although  all  may  be 
represented  by  a single  "Engine  Quits"  situation,  different  courses  of 
action  must  be  taken  upon  the  occurrence  of  different  situations.  There- 
fore, this  event  must  be  divided  into  three  possible  situations  (Figure 
10).  However,  in  such  a case,  assignment  of  subjective  probabilities  of 
occurrence  to  the  three  situations  will  not  be  helpful  since  at  the  time 
of  actual  occurrence  of  this  situation,  the  pilot  knows  which  of  the 
three  situations  has  occurred  and  will  proceed  on  the  branch  representing 
that  situation,  regardless  of  the  probabilities.  In  a way,  a situation 
node  may  be  viewed  as  an  event  node  in  which  the  probabilities  of  different 
branches  emerging  from  it  can  not  be  assessed  successfully  at  the  time  of 
decision  analysis.  However,  at  the  actual  decision  time,  just  before  the 
selection  of  the  course  of  action,  the  probability  of  one  of  the  branches 
becomes  one  and  the  rest  will  be  zero.  In  other  words,  a simple  observa- 
tion will  provide  a masking  scheme  for  the  pilot  to  ignore  the  irrelevant 
portions  of  the  tree  and  concentrate  only  on  the  subtree  emerging  from 
the  branch  representing  the  current  situation.  A simple  example  of  a 
decision  tree  appears  in  Figure  10. 

DECISION  ELEMENT  IDENTIFICATION 

A detailed  analysis  of  the  relevant  workbooks  and  tape  slides  as  well 
as  expert  interviews  identified  the  decision  elements  involved  in  the 
selected  decision  task  areas.  The  alternative  courses  of  action  for  each 
decision  and  the  possible  outcomes  for  each  action  were  specified.  The 
result  was  a large  amount  of  data  which  was  incorporated  in  the  construc- 
tion of  preliminary  decision  trees  for  the  decision  task  areas. 

DECISION  TREE  ELICITATION 

Bases  on  the  results  of  the  detailed  analysis  of  the  selected  decision 
tasks,  a preliminary  decision  tree  was  constructed  for  each  of  the  decision 
tasks.  These  decision  trees  represent  the  relevant  academic  knowledge 
required  for  decision  making  in  the  selected  areas.  The  structured  decision 
tree  representation  provides  a suitable  perspective  for  inspecting  these 
situations  in  terms  of  decision  analysis.  However,  the  development  of  the 
complete  decision  tree  depends  on  the  elicitation  of  non-academic  pieces  of 
knowledge  from  the  experts.  These  pieces  of  knowledge  result  frum  exper- 
ience, and  since  there  is  no  formal  way  of  teaching  such  items,  they  are 
usually  not  included  in  training  materials.  The  process  of  decision  tree 
elicitation  from  experts  can  capture  such  knowledge  and  represent  it  in  a 
formal  structure. 

To  implement  this  approach,  a set  of  scenarios  and  a series  of 
questionnai res  were  developed  and  used  to  elicit  the  complete  decision 
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P1  and  P^:  Probabilities  si:ch  that  P^  + = 1 

and  Utilities 


Figure  10.  Example  Of  A Decision  Tree  Structure 
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trees.  The  scenarios  were  used  in  protocol  analyses  to  modify  the 
structure  of  the  trees,  and  the  questionnaires  were  utilized  in  expert 
interviews  to  provide  probabilities  of  different  outcomes  as  well  as  the 
utilities  of  the  terminal  nodes.  The  protocol  analyses  and  the  expert 
interviews  are  described  in  the  following  subsections.  The  complete 
decision  trees  for  "engine  failure"  and  "land  or  wave  off"  appear  in 
Appendices  F and  G,  respectively. 

PROTOCOL  ANALYSIS.  A protocol  analysis  was  performed  to  check  the  structure 
of  the  preliminary  decision  tree.  A scenario  was  developed  to  acquire 
alternative  courses  of  action  at  each  decision  node,  and  possible  outcomes 
at  each  event  node.  A sample  scenario  appears  in  Table  9.  The  complete 
list  of  scenarios  and  the  questions  to  be  answered  by  the  protocol  analysis 
is  included  in  Appendix  H.  Four  LAMPS  ASW  instructors  were  participants 
in  the  protocol  analyses.  The  results  of  the  analyses  provided  the 
experience-oriented  knowledge  which  was  utilized  to  modify  the  structure 
of  the  preliminary  decision  trees. 

EXPERT  INTERVIEW.  Once  the  structures  of  the  decision  trees  were  completed, 
probabil ities  of  different  outcomes  and  utilities  of  different  terminal 
nodes  were  judgmentally  assessed.  To  accomplish  this  objective,  four  LAMPS 
ASW  instructors  were  interviewed.  To  simplify  the  process  of  eliciting  sub- 
jective probabilities  the  participants  were  asked  to  assign  a number  to 
each  outcome  such  that  the  number  represented  the  relative  chance  of 
occurrence  of  the  corresponding  outcome.  In  this  case,  the  participants 
were  not  obliged  to  provide  probabilities  which  add  up  to  one.  Therefore, 
the  elicitation  process  proceeded  more  efficiently.  Since  the  protocol 
analyses  had  provided  the  exhaustive  list  of  all  possible  outcomes  at  each 
event  node,  a simple  normalization  procedure,  using  chances  of  outcomes 
emerging  from  each  event  node,  calculated  the  final  probabilities  of  the 
outcomes.  To  provide  further  simplification,  the  participants  were  asked 
to  order  the  outcomes  emerging  from  each  event  node  according  to  their 
chance  of  occurrence  prior  to  actual  assessment  of  the  probabilities.  With 
such  a procedure,  the  process  of  probability  assessment  was  performed  in 
an  informal  manner,  and  without  too  much  emphasis  on  the  theoretical  notion 
of  probabilities.  Table  10  represents  a sample  questionnaire  for  probabil- 
ity assessment.  The  results  of  the  probability  assessment  are  indicated 
on  the  decision  trees  in  Appendices  F and  G. 

Five  major  elements  were  Jent-^ied  as  basic  attributes  for  the  terminal 
outcomes.  These  include  (1)  to_al  damage  to  the  aircraft,  (2)  partial 
damage  to  the  aircraft,  (3)  loss  of  pilot  life,  (4)  loss  of  sensor  operator's 
life,  and  (5)  loss  incurred  from  mission  failure.  At  each  terminal  node, 
an  informal  multiattribute  utility  analysis  was  performed  on  the  five 
attributes.  To  perform  such  analysis,  the  degree  of  severity  of  each 
attribute  was  elicited  from  each  participant.  The  questionnaire  used  for 
this  process  appears  in  Table  11.  In  these  situations,  the  assessment  of 
the  degree  of  aversion  is  more  natural  than  the  assessment  of  the  degree 
of  desirability.  Therefore,  instead  of  utilities,  the  losses  incurred  by 
different  attributes  were  assessed.  The  results  of  this  assessment  by  the 
four  participants  appear  in  Table  12.  The  degree  of  aversion  for  each  of 
* these  situations  is  different  in  wartime  compared  to  a training  environment 


53 


f 


NAVTRAEQUIPCEN  77-C-0005-1 


TABLE  9.  SAMPLE  SCENARIO  FOR  PROTOCOL  ANALYSIS 


Consider  yourself  engaged  in  each  of  the  following 
scenarios.  Describe  what  might  happen  in  terms  of 
the  possible  alternative  courses  of  action,  possible 
states  of  nature,  and  the  outcomes  associated  with 
each  combination  of  a state  of  nature  and  a course 
of  action. 

. You  have  a single  engine  failure 
while  flying  over  land.  One  engine 
is  sufficient  for  the  weight  of  the 
aircraft  and  there  is  a safe  site 
in  the  area. 
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TABLE  10.  SAMPLE  QUESTIONNAIRE  FOR  PROBABILITY  ASSESSMENT 


We  want  to  know  the  probability  that  each  outcome  occurs.  You  are  asked 
to  consider  each  outcome  and  tell  me  how  often  you  think  these  situations 
occur.  For  example,  in  the  land  or  wave  off  situation,  you  have  decided 
on  a running  landing:  under  these  conditions,  how  often  are  you  going 
too  fast,  how  often  is  the  sink  rate  too  high,  how  often  is  it  OK? 

Please  use  this  scale  for  each  event  and  try  to  give  me  a number  to 
represent  the  relative  chance  of  occurrence  of  each  event. 


0 10  20  30  40  50  60  70  80  90  100 

| 1 1 1 1 1 1 1 1 1 1 
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TABLE  11.  LOSS  ASSESSMENT  FOR  MAJOR  OUTCOME  ATTRIBUTES 


Determine  the  value  for  each  of  the  following  components 
of  loss  on  a scale  from  0 to  1000. 

1.  A/C  total  damage 

2.  A/C  partial  damage 

3.  Loss  of  pi  lot's  1 ife 

4.  Loss  of  sensor  operator's  life 

5.  Loss  incurred  from  mission  failure 

> 
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TABLE  12.  THE  RESULTS  OF  LOSS  ASSESSMENT  FOR 
THE  FIVE  MAJOR  OUTCOME  ATTRIBUTES 


1 . Loss  of  Pilot's  Life 

2.  Loss  of  Sensor  Operator's  Life 

3.  Loss  of  Mission 

a.  Major 

b.  Minor 

4.  A/C  Total  Damage 

5.  A/C  Partial  Damage 


3b  5 4 3a  2 1 

K 1 1 1 i 1 1 1 1 1 1 

0 100  200  300  400  500  600  700  800  900  1000 

2 

3 5 4 1 

h- 1 1 1 1 1 1 1 1 1 1 

0 100  200  300  400  500  600  700  800  900  1000 

2 

3 5 4 1 

I 1 1 1 1 j 1 1 1 l 1 

0 100  200  300  400  500  600  700  800  900  1000 

2 

5 3 4 1 

(— 1 1 1 1 1 1 1 1 1 1 

0 100  200  300  400  500  600  700  800  900  1000 
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in  peacetime.  Since  the  participants  were  the  experts  in  a peace- 
time training  environment,  the  losses  were  assessed  for  a peacetime 
condition.  The  results  of  the  loss  assessment  also  indicated  the 
differences  among  different  participants'  scales.  This  information  was 
incorporated  in  the  aggregation  of  the  losses  assigned  by  different  par- 
ticipants to  the  same  outcome.  A sample  questionnaire  for  expert  interview 
is  presented  in  Table  13.  The  results  of  the  loss  assessments  are  indicated 
on  the  decision  trees  of  Appendices  F and  G. 
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TABLE  13.  SAMPLE  QUESTIONNAIRE  FOR  EXPERT  INTERVIEW 


In  a training  environment  (during  peacetime  ) when  one  engine  is  not 
sufficient  for  the  total  weight  of  the  aircraft,  what  is  the  chance 
that  after  jettisoning: 

A)  The  aircraft  will  be  light  enough  so  that  one  engine 

is  sufficient?  

B)  The  aircraft  will  not  be  light  enough  so  that  one 

engine  is  sufficient?  

A dual  engine  failure  occurs  when  flying  over  water.  How  do  you 
assess  the  losses  incurred  from  each  of  the  following  possible 
outcomes  on  a 0 to  1000  scale: 

A)  Loss  of  Aircraft?  

B)  Loss  of  Aircraft  and  Injury?  

C)  Loss  of  Aircraft  and  Loss  of  Life?  
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INSTRUCTIONAL  GUIDELINE  DEVELOPMENT 

4 


GENERAL 

The  analysis  of  the  selected  decision  tasks  identified  the  characteristics 
of  decision  tasks  involved  in  operational  systems.  Based  on  this  information, 
a system  for  decision  training  guideline  generation  was  designed  to  be  used  as 
a development  aid  for  decision  training  programs  in  operational  systems.  The 
design  objective  was  to  develop  a system  to  provide  instructional  guidelines 
presented  in  a form  suitable  for  incorporation  into  the  specification  of 
instructional  objectives,  the  planning  of  strategies,  the  selection  of  media, 
and  the  other  steps  required  for  the  production  of  instructional  materials  for 
decision  tasks.  The  system  uses  quantification  schemes  to  incorporate  expert 
knowledge  into  the  guideline  generation  process.  A systematic,  step-by-step 
process  identifies  the  system  functions  of  the  decision  training  guideline 
generator.  The  process  involves  (1)  decision  task  identification,  (2)  decision 
type  recognition,  (3)  decision  task  classification,  (4)  instructional  content 
generation,  (5)  cognitive  level  weighting,  (6)  instructional  method  selection, 
(7)  method/media  relevance  detection,  and  (8)  instructional  media  selection. 

A description  of  the  methodology,  as  well  as  the  modules  of  the  decision 
training  guideline  generation  system,  is  the  subject  of  the  following: 

METHODOLOGY 

Current  efforts  in  the  development  of  decision  training  systems  have 
usually  evolved  in  the  context  of  task-specific  systems  (e.g.,  medical, 
tactical,  managerial,  etc.).  A training  system  for  decision  makers  that  is 
developed  within  the  context  of  such  a task-specific  domain,  without  incor- 
poration of  general  purpose  decision  training  components,  does  not  provide 
trainees  with  the  skills  required  for  making  task-specific  decisions  within 
other  domains.  Furthermore,  since  the  characteristics  of  decision  tasks 
within  any  specific  domain  may  be  subject  to  change  with  time,  such  training 
systems  will  not  even  provide  useful  long-term  guidelines  within  the  trainee's 
own  domain.  In  these  systems,  the  decision  maker  learns  a tailor-made  set  of 
conceptual  components,  procedures,  techniques,  heuristic  rules,  parameters, 
and  values  unique  to  that  particular  system  at  the  time  of  training  (F-gure 
11).  The  set  of  concepts  that  the  potential  decision  maker  learns  is  in 
reality  a very  specific  subset  of  more  general  conceptual  components, 
procedures,  and  techniques.  Yet,  he  never  becomes  aware  that,  in  fact,  it  is 
a subset  of  the  more  general  set  because  it  is  outside  the  training  boundary. 
Hence,  if  a decision  maker  is  transferred  to  a new  decision  making  domain,  or 
the  characteristics  of  the  decision  tasks  within  his  decision  domain  change, 
he  typically  has  two  choices:  he  must  either  enter  a new  task-specific 
decision  training  program  (and  learn  another  unique  conceptual  subset  again 
without  realizing  its  relationship  to  the  first  subset  learned),  or  he  must 
learn  to  invent  his  own  conceptual  set  without  the  benefit  of  a formal 
training  program.  In  the  latter  case,  the  decision  maker  is  without  a formal 
guide,  and  often  attempts  to  "cram"  his  new  world  into  a conceptual  subset 
learned  for  an  entirely  different  decision  task. 


60 


NAVTRAEQUIPCEN  77-C-0005-1 


Degree  of  Generality 


Highest 


Lowest 


Top  Down  Approach 


General-Purpose 


Task-Spec Ific 


Bottom  Up  Approach 


CONCEPTUAL 

COMPONENTS 

PROCEDURES 

TECHNIQUES 

HEURISTIC 

RULES 

PARAMETERS 

NOMENCLATURE 

PARAMETER 

QUANTIFICATION 


PROBLEM  RECOGNITION.  INFORMATION  ACQUISITION 
ALTERNATIVE  DEVELOPMENT,  ETC. 

TRANSITION  RULES.  PROCEDURES  ORDERING. 
SUBTASKS  CONSIDERATION,  ETC. 

AMETER  ASSESSMENTS.  PROBLEM  STRUCTURING, 
DECISION  TREES  DEVELOPMENT,  ETC. 

PLAUSIBLE  ALTERNATIVES,  IMPACTS,  PROBLEMS 
CLASSIFICATION,  ETC. 

VALUE  DEFINITION,  EVENT  SPECIFICATIONS, 
PROPER  CRITERIA 

PARAMETERS  OOMAIN,  SPECIFIC  PROBABILITIES 
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Figure  11.  Levels  of  Decision  Training 
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In  the  development  of  the  decision  training  guideline  generation 
system,  transferability  was  one  of  the  basic  design  considerations.  Al- 
though the  project  has  been  initially  aimed  to  develop  a systematic  pro- 
cedure for  providing  instructional  guidelines  for  decision  tasks  encountered 
in  LAMPS  ASW  Operation,  a generalized  methodology  was  developed  to  underlie 
the  design  in  order  to  make  the  application  of  the  system  to  other  task 
domains  subject  to  minimal  modification  effort. 

The  project  started  with  an  analysis  of  tasks  involved  in  the  operator's 
job.  Decision  tasks  were  identified  and  analyzed.  The  result  of  the 
analysis  was  a detailed  decision  tree  for  each  decision  task  which  provided 
the  task-specific  knowledge.  For  each  task,  the  parameters  involved  were 
identified  and  quantified.  Heuristic  rules  and  techniques  relevant  to 
decision  making  process  for  the  tasks  were  selected  and  considered  for  the 
training  program.  However,  a training  system  development  approach  based 
solely  on  the  above  task  specific  knowledge,  without  incorporation  of 
general  purpose  decision  training  components,  will  limit  the  scope  of  the 
system.  To  avoid  this  problem  and  to  provide  for  adaptability  to  new 
decision  situations,  information  concerning  general  decision  making  be- 
havioral objectives  were  included  in  the  design  of  the  system. 

DECISION  TRAINING  BEHAVIORAL  OBJECTIVES.  Although  a task  specific  decision 
training  system  is  built  around  a nucleus  of  task  specific  knowledge,  its 
primary  objective  is  to  create  the  required  decision  making  "behaviors"  in 
trainees.  Since  there  is  a common  underlying  set  of  skills  required  for 
performing  every  decision  making  task,  a set  of  general  decision  making 
behaviors,  associated  with  those  skills,  can  be  identified  as  behavioral 
objectives  for  any  decision  training  system.  These  behavioral  objectives 
will  provide  the  required  general  purpose  decision  training  components  for 
the  system.  Identification  of  such  decision  training  behavioral  objec- 
tives is  the  subject  of  this  section. 

Different  structures  of  conceptualizing  decision  processes  have  been 
proposed.  Some  conceptualizations  emphasize  differences  among  decision 
situations;  other  focus  on  the  tasks  that  decision  makers  are  required 
to  perform.  Nickerson  and  Ferhrer  (1975)  present  an  overview  of  these 
structures . 

Adelson  (1961)  proposed  a taxonomy  of  decision  tasks  that  are  carried 
out  in  modern  command  and  control  systems: 

a-  Characterization  of  the  state  of  the  world 

b-  Determination  of  the  available  action  alternatives 

c*  Outcome  prediction 

d*  Choice  rationalization 

Six  steps  for  a business  executive  decision  making  process  have  been 
identified  by  Drucker  (1967)  as: 

a.  Classification  of  the  problem 

b.  Definition  of  the  problem 
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c-  Specifications  that  the  answer  to  the  problem  must  satisfy 
d-  Decision  as  to  what  is  right 

e-  Building  into  the  decision,  the  action  to  carry  it  out 

f-  Feedback  which  tests  the  validity  and  effectiveness  of  the 
decision  against  actual  course  of  events. 

The  following  thirteen  steps  are  suggested  by  Edwards  (1965)  as  different 
categories  of  activities  that  must  be  carried  out  by  any  Bayesian  decision 
maker: 

a-  Recognition  of  the  existence  of  a decision  problem 
b-  Identification  of  available  actions 

c.  Identification  of  relevant  states  that  determine  payoff  for 
actions 

d-  Identification  of  the  value  dimensions  to  be  aggregated  into  the 
payoff  matrix 

e-  Judgmental  evaluation  of  each  outcome  and  each  dimension 

f • Aggregation  of  value  judgments  into  a composite  payoff  matrix 

g.  Identification  of  information  sources  relevant  to  discrimination 
among  states 

h-  Data  collected  from  information  sources 
i . Data  f i 1 tration 
j.  Estimation  of  likelihood  ratios 
k-  Aggregation  of  impact  estimates 

1*  Selection  of  best  alternative  by  expected  value  maximization 
m-  Implementation  of  the  decision. 

Hill  and  Martin  (1971)  have  recognized  six  different  steps  involved  in 
the  decision  making  process. 

a-  Identification  of  concern 

b-  Diagnosis  of  situation 

c-  Formulation  of  action 

d-  Test  of  feasibility  of  selected  alternatives 
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e.  Adoption  of  alternative 

f . Assessment  of  consequences  of  adopted  alternative. 

A more  structured  conceptualization  of  the  decision  making  process 
has  been  performed  by  Schrenk  (1969): 

a.  Problem  recognition,  involving: 

Information  acquisition 
. Objective  recognition 
. Decision  need  perception 
. Problem  urgency  and  importance  assessment 

b . Problem  diagnosis,  involving: 

. Definition  of  possible  situations 
. Evaluation  of  situation  likelihoods 

Determination  of  whether  more  information  is  needed 
Identification  of  possible  data  sources 
. Judgment  of  value  versus  cost 
Seeking  of  information 
Diagnosis  of  decision  situation 
. Determination  of  whether  alternatives  under  consideration 
account  for  all  the  data 
. Diagnostic  decision. 

c.  Action  Selection,  involving: 

. Definition  of  action  goals 
. Specification  of  value  and  time  criteria 
. Weighting  of  decision  criteria 
. Specification  of  risk  philosophy 
. Consideration  of  operating  doctrine 
. Generation  of  action  alternatives 
. Prediction  of  possible  outcomes 

Estimation  of  outcome  gains  and  losses 
. Estimation  of  outcome  likelihoods 

Evaluation  of  expected  values  of  action  versus  their  costs 
Evaluation  of  actions  by  risk  philosophy 
. Determination  of  whether  more  information  is  needed 
Seeking  of  information 
. Re-evaluation  of  action  alternatives 
. Determination  of  whether  the  action  if  acceptable 
. Selection  of  course  of  action 
. Implementation  of  course  of  action. 

Soelberg  (1966)  suggests  six  aspects  of  the  decision  making  process  as 

a-  Problem  recognition 
b-  Problem  definition 
c • Planning 
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d . Search 

e.  Confirmation 

f.  Implementation 

An  analysis  of  these  alternative  conceptualizations  reveals  the  conmon 
underlying  structure  existing  in  all  of  them.  This  common  structure  iden- 
tifies a decision  task  to  be  composed  of  either  or  both  of  two  major  sub- 
tasks: problem  structuring  and  alternative  selection.  Each  major  subtask, 
in  turn,  may  be  further  subdivided  into  a number  of  smaller  subtasks. 

Problem  structuring  can  be  divided  into  alternatives  formulation  and  out- 
come identification;  alternative  selection  can  be  partitioned  into  utility 
assessment,  probability  assessment,  decision  rule  application,  and  best 
alternative  selection  (Figure  12).  The  possible  transition  between  sub- 
tasks in  a decision  making  process  is  outlined  in  Figure  13. 

The  six  subtasks  appearing  as  tip  nodes  in  Figure  12  represent  the 
general  decision  making  components  which  must  be  included  in  a decision  train- 
ing program.  These  components  are  restated  as  instructional  objectives: 

a.  Formulate  alternative  courses  of  action  for  a given  decision 
situation. 

b.  Establish  outcomes  of  a given  course  of  action  for  a given 
decision  situation. 

c.  Assess  the  utility  of  a given  outcome  in  a given  decision 
situation. 

d.  Assess  probabilities  of  a given  set  of  events  which  may  occur 
as  a result  of  a given  alternative  course  of  action  in  a given 
decision  situation. 

e.  Apply  a relevant  decision  rule  to  a given  decision  situation. 

f*  Select  the  best  alternative  course  of  action  for  a given 
decision  situation. 

Each  instructional  objective  is  defined  in  terms  of  specific  learning 
outcomes,  that  is,  a representative  sample  of  specific  types  of  behavior 
that  is  to  be  used  as  evidence  that  the  objective  has  been  achieved.  The 
derived  behavioral  objectives  associated  with  each  instructional  objective 
are  shown  in  Table  14. 

INSTRUCTIONAL  GUIDELINE  GENERATION  SYSTEM 

Based  on  the  methodology  developed  in  this  chapter,  a system  was  de- 
signed to  provide  guidelines  for  the  development  of  training  programs  in 
the  area  of  decision  making.  The  guidelines  include  the  suggestion  of 
instructional  contents,  methods,  and  media  compatible  to  the  specific 
decision  task  as  well  as  characteristics,  requirements,  and  resources  of 
the  training  center. 
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Figure  12.  Decision  Making  Subtasks 
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Figure  13.  Transition  Between  Decision  Making  Subtasks 
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TABLE  14.  DECISION  TRAINING  BEHAVIORAL  OBJECTIVES 


1.  Formulate  alternative  courses  of  action  for  a given 

decision  situation. 

1.1  Identi fy  the  variable(s)  and  parameter(s)  in 
the  decision  situation. 

1.2  Classify  each  of  the  variables/parameters  of 
a given  set  as  relevant  or  irrelevant  to  the 
decision  situation. 

1.3  Identify  constraints  on  each  variable/parameter 
based  on  the  statement  of  decision  situation 
and  general  domain  knowledge. 

1.4  Define  solution  range  (set)  of  the  variable(s) 
for  the  decision  situation. 

1.5  Generate  a set  of  alternatives  using  some 
or  all  of  the  variables/parameters. 

1.6  Identify  those  alternatives  which  are  feasible 
in  light  of  constraints  on  individual  variables, 
statement  of  decision  situation,  and  general 
domain  knowledge. 
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TABLE  14.  DECISION  TRAINING  BEHAVIORAL  OBJECTIVES  (CONTINUED) 


2.  Establish  outcomes  of  a given  course  of  action  for  a given 

decision  situation. 

2.1  State  the  possible  events  that  may  occur  as  a result 
of  application  of  the  alternative  course  of  action  to 
the  decision  situation. 

2.2  Identify  the  variables  in  the  decision  situation 
that  will  be  affected  if  a particular  event  occurs 
as  a result  of  application  of  the  alternative 
course  of  action  to  the  decision  situation. 

2.3  State  the  resulting  decision  situation  in  terms  of 
the  variables  when  a particular  event  occurs  as  a 
result  of  application  of  the  alternative  course  of 
action  to  the  decision  situation. 
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TABLE  14.  DECISION  TRAINING  BEHAVIORAL  OBJECTIVES  (CONTINUED) 


3.  Assess  the  utility  of  a given  outcome  in  a given  decision 

situation. 

3.1  State  the  possible  external  effects  of  the  outcome 
on  the  decision  situation. 

3.2  State  the  likelihood  that  a given  external  effect 
wi 1 1 result  from  the  given  outcome  in  the  decision 
situation. 

3.3  Identify  whether  a given  effort  of  the  outcome  is 
favorable  or  unfavorable  for  the  decision  situation. 

3.4  State  the  degree  of  importance  of  a given  external 
effect  resulted  from  the  outcome  from  the  decision 
situation. 

3.5  Estimate  the  aggregate  utility  of  the  outcome  based 
on  the  given  possible  external  effects  with  given 
chances  of  occurrence,  given  direction  of  effect 
(negative  effect  or  positive  effect),  and  given 
magnitude  of  importance. 
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TABLE  14.  DECISION  TRAINING  BEHAVIORAL  OBJECTIVES  (CONTINUED) 


4.  Assess  probabilities  of  a given  set  of  events  which 
probably  will  occur  as  a result  of  a given  alternative 
course  of  action  in  a given  decision  situation. 

4.1  Identify  the  event  with  the  highest  chance  of 
occurrence  among  the  set  of  events. 

4.2  Sort  the  set  of  events  according  to  the  order  of 
likelihood  of  occurrence  in  the  decision  situation. 

4.3  Assign  likelihood  values  to  the  given  ordered  list 
of  events  in  relation  to  the  decision  situation. 

4.4  Calculate  probabilities  of  events  for  the  given 
ordered  list  of  events  with  given  likelihood 
values  by  normalizing  the  likelihood  values. 
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TABLE  14.  DECISION  TRAINING  BEHAVIORAL  OBJECTIVES  (CONTINUED) 


5.  Apply  a relevant  decision  rule  to  a given  decision 

situation. 

5.1  Name  all  decision  rules. 

5.2  Describe  the  procedure  for  applying  each 
decision  rule. 

5.3  State  the  advantages  and  disadvantages  involved 
in  application  of  each  decision  rule. 

5.4  Define  the  criteria  and  procedure  for  selection 
of  a decision  rule  based  on  the  characteristics 
of  the  given  decision  situation  and  advantages/ 
disadvantages  of  the  available  decision  rules. 

5.5  Select  the  best  decision  rule  for  the  given 
decision  problem  by  comparing  each  decision 
rule  in  terms  of  the  criteria  defined  and 
selecting  the  most  desirable  one  on  the  basis 
of  relative  merits  with  respect  to  the  decision 
problem. 

5.6  Apply  a given  decision  rule  to  the  given  decision 
problem. 
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TABLE  14.  DECISION  TRAINING  BEHAVIORAL  OBJECTIVES  (CONTINUED) 


6.  Select  the  best  alternative  course  of  action  for  a 

given  decision  situation. 

6.1  Name  all  selection  criteria  for  the  best  alternative. 

6.2  State  selection  criteria  relevant  to  a given 
decision  rule. 

6.3  Select  the  best  selection  criterion  for  the 
given  decision  situation,  based  on  a given 
decision  rule. 

6.4  Applx  a given  selection  criterion  to  the  given 
decision  situation  after  the  application  of  a 
given  decision  rule. 
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The  instructional  guideline  generation  system  is  basically  composed 
of  two  modules:  Decision  Task  Classification,  and  Decision  Training  Guide- 
line Generator  (Figure  14).  A task,  which  has  been  already  identified  as  a 
decision  task,  enters  the  Decision  Task  Classification  module.  This 
module  acts  as  a preprocessor  which  associates  the  task  with  one  of  the 
decision  classes  relevant  to  the  training  objectives.  The  output  will  be 
class  information  which  enters  the  Decision  Training  Guideline  Generator 
Module.  Based  on  the  classification  result,  as  well  as  training  environ- 
ment characteristics,  requirements,  and  resources,  the  module  suggests 
the  required  instructional  content  and  the  best  instructional  methods  and 
media  for  the  input  decision  task. 

A detailed  diagram  describing  the  relationships  among  different  com- 
ponents of  the  system  is  shown  in  Figure  15.  A brief  description  of  the 
eight  components  and  their  interactions  as  well  as  the  input/output  of  the 
system  is  described  in  the  remainder  of  this  section.  In  what  follows, 
attention  must  be  paid  to  the  fact  that  the  implicit  procedure  defined  by 
the  system  is  being  performed  (operated)  by  an  instructional  technologist. 

Decision  Task  Identification  receives  any  training  task  presented  in 
the  objective  hierarchies  and  works  as  a filter  which  separates  decision 
tasks  from  non-decision  tasks.  The  specification  of  this  component  follows 
the  definition  of  a decision  task: 

a.  The  objective  of  a decision  task  is  to  select  an  alternative 
from  a specified  set  of  al ternati ves. 

b.  This  selection  may  require  the  formulation  of  alternatives 
(problem  structuring). 

c.  There  is  a lack  of  completely  specified  criteria  for  either 
alternative  formulation  or  alternative  selection. 

The  output  of  the  component  is  the  identification  of  the  input  task 
either  as  a decision  task,  which  causes  the  task  to  enter  the  Decision  Type 
Recognition  and  Decision  Task  Classification  components,  or  as  a non- 
decision task,  which  results  the  initiation  of  a conventional  ISD  procedure. 

Decision  Type  Recognition  receives  a decision  task  and  identifies  it 
as  Type  1,  Type  2,  or  Type  3.  Type  1 represents  decision  tasks  with 
processes  requiring  only  problem  structuring;  Type  2 includes  all  decision 
tasks  concerning  only  action  selection;  Type  3 is  composed  of  all  decision 
tasks  which  require  both  problem  structuring  and  action  selection.  The 
information  concerning  the  type  of  the  decision  task  enters  Decision  Task 
Classification  and  Cognitive  Level  Weighting  components. 

Decision  Task  Classification  is  based  on  a procedure  similar  to  that 
of  discriminant  analysis.  To  distinguish  between  classes  of  decision  tasks, 
a collection  of  discriminating  variables  is  selected  that  measures  charac- 
teristics on  which  the  classes  are  expected  to  differ,  that  is,  training 
content  requirements.  Thirty  different  classes  constitute  the  set  of 
possible  classes  relevant  to  decision  training.  The  decision  task  and 
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Figure  14.  Overview  of  Instructional  Guideline  Generation  System 
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the  information  concerning  its  type  enter  the  Decision  Task  Classification 
component.  The  output  of  the  component  represents  the  class  of  the  deci- 
sion task  which  is  sent  to  the  Instructional  Content  Generation  component. 

Instructional  Content  Generation  receives  the  information  about  the 
class  of  the  decision  task  and  generates  the  reguired  instructional  contents. 
Since  a set  of  basic  instructional  contents  is  associated  with  each  class, 
the  process  involved  is  a simple  table  look-up.  The  basic  instructional 
content  appears  as  part  of  the  system's  output  which  together  with  the  task- 
specific  information  presented  in  decision  trees,  constitute  required 
instructional  content  for  the  decision  training  program. 

Cognitive  Level  Weighting  receives  the  decision  type  and  selects  a 
cognitive  level  weight  vector.  The  idea  of  a cognitive  level  weight  vector 
was  initiated  by  the  fact  that  the  training  objectives  for  different  tasks 
with  unequal  cognitive  complexity  can  be  accomplished  differently  depending 
on  the  instructional  method  employed.  Such  dependency  between  cognitive 
complexity  and  instructional  method  suggested  a detailed  analysis  of  cog- 
nitive levels  involved  in  the  performance  of  each  decision  behavior.  The 
result  of  the  analysis  was  the  identification  of  a cognitive  level  weight 
vector  associated  with  each  decision  type.  Each  vector  presents  the  degree 
of  importance  of  the  six  cognitive  levels  with  respect  to  the  corresponding 
decision  type.  The  six  cognitive  levels  include:  knowledge,  comprehension, 
application,  analysis,  synthesis,  and  evaluation.  Since  a weight  vector  is 
assigned  to  each  decision  type,  the  process  involved  is  a simple  table 
look-up.  The  selected  cognitive  level  weight  vector  is  sent  to  Instruc- 
tional Method  Selection  component. 

Instructional  Method  Selection  receives  the  cognitive  level  weight 
vector,  incorporates  it  with  the  data  in  the  method/attribute  matrix,  and 
results  in  the  most  effective  instructional  method.  The  method/attribute 
matrix  represents  the  judgmental  evaluation  of  the  degree  of  effectiveness 
of  fourteen  major  instructional  methods  with  respect  to  the  six  cognitive 
levels.  The  judgmental  data  was  provided  by  a training  specialist  and  a 
psychologist.  A weighted  average  procedure  is  used  to  calculate  a degree 
of  effectiveness  for  each  instructional  method  with  respect  to  the  deci- 
sion task.  The  method  with  the  highest  degree  of  effectiveness  is  selected 
as  the  most  effective  instructional  method  and  appears  as  part  of  the 
system's  output. 

Method/Media  Relevancy  Detection  receives  the  selected  instructional 
method  and,  usTng  the  method7medi  a relevancy  matrix,  identifies  the  media 
relevant  to  the  selected  method.  Since  some  instructional  media  are  not 
applicable  to  a training  environment  employing  a specific  instructional 
method,  a study  of  the  relevancy  of  each  of  the  twelve  instructional  media 
with  respect  to  each  instructional  method  was  performed.  The  result  of  the 
study  dichotomized  the  relevant  and  irrelevant  media  for  each  instructional 
method.  These  results  appear  in  the  method/media  relevancy  matrix.  The 
relevant  instructional  media  are  identified  by  a reference  to  the  method/ 
media  relevancy  matrix,  and  the  information  is  sent  to  the  Instructional 
Media  Selection  component. 
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Instructional  Media  Selection  receives  the  relevant  media  and,  using 
the  media/attribute  matrix  and  the  weights  of  different  attributes  with  respect 
to  the  training  environment,  selects  the  most  effective  media.  The  degree 
of  effectiveness  of  each  instructional  media  with  respect  to  different 
attributes  was  judgmental ly  assessed  by  a training  specialist.  During 
system's  application,  the  system  user  instructional  technologist  will  provide 
a weight  vector  which  identified  the  degree  of  importance  of  different 
attributes  of  the  instructional  method  with  respect  to  the  specific  training 
environment.  Then,  a weighted  average  procedure  will  be  used  to  calculate 
the  degree  of  effectiveness  of  each  relevant  instructional  media  with  respect 
to  the  decision  task.  The  media  with  the  highest  degree  of  effectiveness  is 
selected  as  the  most  effective  instructional  media  and  appears  as  part  of 
the  system's  output. 

DECISION  TASK  CLASSIFICATION  PREPROCESSOR 

The  decision  task  space  can  be  divided  into  different  classes  such  that 
each  class  represents  a set  of  decision  tasks  with  some  degree  of  similarity 
with  respect  to  their  training  requirements.  The  inter-class  similarity 
will  then  suggest  implementation  of  similar  instructional  content,  method, 
and  media  for  all  decision  tasks  belonging  to  the  same  class.  The  classifica- 
tion scheme  must  be  oriented  toward  training  and  its  discriminant  elements 
must  be  designed  based  on  decision  task  differences  which  suggest  different 
training  strategies. 

Such  a classification  scheme  was  designed  with  two  components:  decision 
type  recognition  and  decision  task  classification.  Decision  type  recognition 
is  based  on  the  methodology  described  in  Chapter  2.  Three  different  types  of 
decision  tasks  are  identified.  Type  1 represents  decision  tasks  with 
processes  requiring  only  problem  structuring;  type  2 includes  all  decision 
tasks  concerning  only  action  selection;  type  3 is  composed  of  all  decision 
tasks  which  require  both  problem  structuring  and  action  selection.  Since 
the  processes  involved  in  problem  structuring  are  of  different  cognitive 
complexity  than  the  ones  involved  in  action  selection,  the  training  processes 
for  the  tasks  belonging  to  different  decision  types  must  be  conducted  differ- 
ently. Such  differences  in  training  processes  will  be  specified  in  terms  of 
the  guidelines  for  instructional  content,  method,  and  media. 

In  constructing  the  second  component,  decision  task  classification,  the 
objective  is  to  provide  a procedure  which  assigns  each  decision  task  to  a 
category  associated  with  the  most  relevant  instructional  strategy.  If  all 
decision  tasks  can  be  so  categorized  into  a given  number  of  classes,  and  if 
training  can  be  developed  for  each  class,  then  any  decision  task  may  be 
easily  categorized  and  appropriate  training  programs  easily  developed  for  it. 

This  procedure  is  similar  to  that  of  discriminate  analysis.  Discrimi- 
nant analysis  begins  with  the  desire  to  statistically  distinguish  between 
two  or  more  groups  of  cases  (in  this  instance,  a case  would  be  decision  task 
and  discriminant  analysis  would  be  a means  of  distinguishing  among  two  or 
more  classes  of  decision  tasks).  To  distinguish  between  classes,  a collection 
of  discriminating  variables  is  selected  that  measures  characteristics  on  which 
theTTasses  are  expected  to  differ.  Ideally,  discriminating  variables  are 
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assigned  to  classes  in  such  a way  that  the  separation  of  the  classes  is 
maximized.  A set  of  variables  was  designed  which  provides  satisfactory 
discrimination  for  decision  tasks  with  known  class  membership.  The  classes 
have  been  identified,  and  the  classification  of  any  new  decision  task  with 
unknown  membership  can  be  accomplished. 

The  key  to  developing  a decision  task  classification  scheme  is  the 
identification  of  the  variables  or  decision  task  attributes  to  be  used  as 
discriminating  variables. 

There  are  many  attributes  associated  with  decision  making  tasks.  The 
following  represents  a preliminary  listing  of  the  ones  most  relevant  to  a 
training  program. 

a.  Single  attribute  - multi-attribute 

b • Individual  - group 

c-  Static  - dynamic 

d-  One  shot  - repetitive 

e-  Certainty  - risk/uncertainty 

f.  Abstract  (general)  - concrete  (task  specific) 

g-  Well  defined  - ambiguous 

h-  Decision  making  - decision  execution  (Note:  this  characteristic 
follows  a convention  developed  in  this  project.) 

i-  Time  critical  - time  relaxed 

j-  Small  probability  high  loss  - normal  ranges  for  probability  and  loss 

k-  Type  1 (problem  structuring),  type  2 (alternative  selection), 
and  type  3 (problem  structuring  and  an  alternative  selection). 

(Note:  This  characteristic  follows  a convention  developed  in 
this  project.) 

1.  Decision  rule  (SEU,  EV,  Mini-max,  Maxi-min,  etc.) 

Each  attribute  describes  a different  dimension  of  decision  making  and  represents 
a different  "pigeon  hole"  in  which  a decision  task  could  be  categorized.  Given 
this  set  of  12  attributes,  and  allowing  for  seven  different  and  mutually 
exclusive  types  of  decision  rules  provides  a very  modest  listing  of  decision 
classes,  compared  to  all  possible  classes  that  are  potentially  available  to 
decision  makers.  However,  even  with  such  a list,  the  implications  as  to  the 
number  of  classes  into  which  decision  tasks  could  be  discriminated  is 
enormous.  Clearly,  this  many  classes  is  neither  feasible  nor  desirable.  What 
assumptions  can  be  reasonably  made  to  reduce  this  number  of  classes  to  a more 
manageable  number?  First,  since  we  are  interested  in  training  ATO's  and 
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Pilots  in  the  LAMPS  ASW  operation,  we  can  assume  that  the  decision  tasks, 
and  hence  training,  will  focus  on  individual  decision  makers.  Next,  assume 
that  all  decisions  are  repetitive  decisions.  That  is,  although  a decision 
is  unique  to  a particular  ASW  mission  and  the  parameters  of  that  mission,  an 
outside  observer  would  find  that  there  is  a great  deal  of  similarity  in  the 
decision  tasks  across  all  pilots  and  ATO's  and  across  all  ASW  missions.  For 
example,  the  decision  of  what  sonobuoy  pattern  to  drop  occurs  in  each  and 
every  ASW  mission.  Also,  assume  that  each  decision  is  task  specific  rather 
than  general.  Each  decision  must  contribute  to  overall  mission  effectiveness 
and,  hence,  must  be  task  specific  at  the  point  in  the  mission  at  which  it  is 
made.  Moreover,  assume  that  since  we  are  interested  in  training  decision 
makers,  we  are  not  concerned  here  with  decision  executions  that  are  the 
result  of  previously  made  decisions.  It  is  also  valid  to  assume  that  all 
decisions  are  made  under  risk  and  in  normal  ranges  of  probability  and  loss. 

At  this  point  it  is  useful  to  refer  to  Figure  16  and  Table  15  which  is  a more 
detailed  representati on  of  what  a decision  task  classification  preprocessor 
might  look  like.  Note  that  a decision  task  follows  one  of  three  major  flows: 
(1)  type  1 decision,  (2)  type  2 decision,  or  (3)  type  3 decision.  Types  1 
and  2 are  of  major  interest,  since  type  3 is  merely  a combination  of  types 
1 and  2.  Type  1 focuses  on  problem  structuring.  The  attributes  of  interest 
here  are: 

a.  Single  attribute  - multi-attribute 

b.  Well  defined  - ambiguous 

c . Time  relaxed  - time  pressure 

d . Static  - dynamic 

Since  type  1 is  only  limited  to  problem  structuring,  decision  rules  are  not 
relevant.  The  focus  is  on  alternative  generation  and  outcome  estimation. 
Therefore,  for  the  type  1 flow,  there  are  2x2x2x2=16  possible  classes 
into  which  decision  tasks  may  fall.  For  type  2 decision  tasks,  those  that 
focus  on  alternative  selection,  the  following  attributes  are  relevant: 

a.  Time  relaxed  - time  pressure 

b.  Decision  rules 

. Subjective  expected  utility  (SEU) 

. Maxi-min  utility  (MMN) 

. Maxi-max  utility  (MMX) 

. Mini -max  regret  (MMR) 

. Lexicography  (LEX) 

. Hurwicz  (HUR) 

. Satisfying  (SAT) 

Since  the  focus  here  is  on  alternative  selection,  there  are  2 x 7 = 14  classes 
into  which  type  2 decision  tasks  may  be  sorted.  Type  3 decision  tasks  can 
be  sorted  using  the  type  1 and  type  2 classes  in  conjunction.  There  are, 
therefore,  16  x 24  = 224  classes  into  which  type  three  decision  tasks  may 
be  sorted,  but  since  these  classes  are  simply  combinations  of  type  1 and  type  2 
tasks  they  do  not  need  to  be  separately  identified.  When  classifying  decision 
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TABLE  15.  DECISION  TASK  CLASSES 

DECISION 

CLASS 

NUMBER  TYPE  CLASS  ATTRIBUTE 

1 1 SWRS  Single  attribute,  well  defined,  time  relaxed,  static 

2 1 SWRD  Single  attribute,  well  defined,  time  relaxed,  dynamic 

3 1 SWPS  Single  attribute,  well  defined,  time  pressure,  static 

4 1 SWPD  Single  attribute,  well  defined,  time  pressure,  dynamic 

5 1 SARS  Single  attribute,  ambiguous,  relaxed,  static 

6 1 SARD  Single  attribute,  ambiguous,  relaxed,  dynamic 

7 1 SAPS  Single  attribute,  ambiguous , pressure,  static 

8 1 SAPD  Single  attribute,  ambiguous,  pressure,  dynamic 

9 1 MWRS  Multiattribute,  well  defined,  time  relaxed,  static 

10  1 MWRD  Multiattribute,  well  defined,  time  relaxed,  dynamic 

11  1 MWPS  Multiattribute,  well  defined,  time  pressure,  static 

12  1 MWPD  Multiattribute,  well  defined,  time  pressure,  dynamic 

13  1 MARS  Multiattribute,  ambiguous,  time  relaxed,  static 

14  1 MARD  Multiattribute,  ambiguous,  time  relaxed,  dynamic 

15  1 MAPS  Multiattribute,  ambiguous,  time  pressure,  static 

16  1 MAPD  Multiattribute,  ambiguous,  time  pressure,  dynamic 
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TABLE  15.  DECISION  TASK  CLASSES  (CONTINUED) 


DECISION 

CLASS 


NUMBER 

TYPE 

CLASS 

ATTRIBUTE 

17 

2 

RSEU 

Time 

relaxed,  subjective  expected  utility 

18 

2 

PSUE 

Time 

pressure,  subjective  expected  utility 

19 

2 

RMMN 

Time 

relaxed,  Maxi -min  utility 

20 

2 

PMMN 

Time 

pressure,  Maxi-min  utility 

21 

2 

RMMS 

Time 

relaxed.  Maxi -max  utility 

22 

2 

PMMX 

Time 

pressure,  Maxi -max  utility 

23 

2 

RMMR 

Time 

relaxed.  Mini -max  regret 

24 

2 

PMMR 

Time 

pressure.  Mini-max  regret 

25 

2 

RLEX 

Time 

relaxed.  Lexicography 

26 

2 

PLEX 

Time 

pressure.  Lexicography 

27 

2 

RHUR 

Time 

relaxed,  Hurwicz 

28 

2 

PHUR 

Time 

pressure,  Hurwicz 

29 

2 

RSAT 

Time 

relaxed,  satisfying 

30 

2 

PSAT 

Time 

pressure,  satisfying 
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tasks  in  this  manner,  30  different  classes  are  required  provided  that  the 
decision  tasks  are  initially  presorted  into  type  1,  2,  or  3 and  that  type  3 
decision  tasks  can  be  trained  by  taking  the  relevant  type  1 and  type  2 tasks 
in  conjunction.  Thirty  decision  classes  is  a reasonable  and  manageable  number 
of  classes  for  which  training  materials  can  be  developed  and  tested.  The 
resulting  decision  classes  are  shown  in  Table  15. 

Type  3 decision  tasks  that  enter  the  system  are  diagnosed  serially  through 
the  type  1 and  type  2 decision  classes.  For  example,  a decision  task  that  is 
type  3 could  appear  as  3-MMPS-PMMR  which  is  (1)  multiattribute,  well  defined, 
time  pressure,  static,  for  alternative  generation  and  outcome  estimation,  and 
(2)  time  pressure,  mini-max  regret  for  alternative  selection.  Note  that  some 
combinations  cannot  exist  (or  are  unlikely  to  exist)  such  as  time  pressure  in 
type  1 and  time  relaxed  in  type  2.  To  train  for  type  3 - MWPS-PMMR,  the  system 
will  simply  take  the  training  package  for  1-MWPS  and  the  training  package  for 
2-PMMR  and  form  one  complete  training  package.  The  structure  of  the  decision 
task  space  for  training,  resulting  from  the  classification  scheme,  is  shown  in 
Figure  17. 

INSTRUCTIONAL  CONTENT  GENERATOR 

Identification  of  different  implications  of  each  decision  class  for 
training  purposes  is  a task  which  must  ollow  the  development  of  the 
classification  scheme.  Such  implications  must  reveal  the  basic  content 
requirements  for  each  class.  Identification  of  these  implications  is  the 
subject  of  the  following.  The  training  implications  of  each  class  are 
represented  in  terms  of  a set  of  modules  corresponding  to  possible  attributes 
of  a decision  task  defined  for  the  task  classification  scheme. 

The  basic  instructional  content  of  each  class  will  be  a combination  of 
the  contents  associated  with  the  attributes  of  that  class.  However,  in  a task- 
specific  decision  training  context,  such  as  the  present  program,  the 
instructional  content  for  each  class  will  be  affected  by  the  task-specific 
information  which  serves  as  a vehicle  for  conveying  the  basic  content.  Such 
task-specific  information  is  provided  by  decision  trees  constructed  for  each 
decision  situation.  For  example,  if  utility  is  a basic  training  content  and 
the  task  is  sonobuoy  patterns,  then  the  utility  of  sonobuoy  patterns  must  be 
included  in  the  instructional  content. 

During  this  program,  the  implications  of  the  relevant  decision  task 
attributes  and  decision  rules  were  analyzed.  The  analysis  emphasized  the 
training  aspects  and  included  the  selected  eight  attributes  (Single  Attribute, 
Multiattribute,  Well  Defined,  Ambiguous,  Time  Relaxed,  Time  Pressured,  Static, 
and  Dynamic)  and  the  seven  decision  rules  (Subjective  Expected  Utility,  Maxi- 
min  Utility,  Maxi-max  Utility,  Mini-max  Regret,  Lexicography,  Hurwicz,  and 
Satisfying).  The  result  was  the  identification  of  the  basic  instructional 
content  required  for  each  decision  attribute.  This  information  was  structured 
in  fifteen  modules,  each  module  represents  information  concerning  General 
Description,  Amplification,  Rules,  Pitfalls/Limitations,  Interacting  wi th 
Other  Decision  Elements,  and  Prerequisites  associated  with  the  corresponding 
attribute  or  decision  rule.  Due  to  the  large  volume  of  the  information,  and  to 
avoid  the  disparity  of  thought,  the  results  are  presented  in  Appendix  J. 
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COGNITIVE  LEVEL  WEIGHTING 

The  training  objectives  for  different  tasks  with  unequal  cognitive 
complexity  can  be  accomplished  differently  depending  on  the  instructional 
methods  employed.  Such  dependency  between  cognitive  complexity  and  instruc- 
tional method  suggests  a detailed  analysis  of  cognitive  levels  involved  in 
performance  of  each  decision  behavior  required  for  different  decision  making 
components . 

Such  an  analysis  was  performed  to  identify  the  highest  cognitive  level 
involved  in  the  decision  making  behaviors.  Six  cognitive  levels  are  con- 
sidered as  the  major  levels  relevant  to  the  process  of  decision  training: 
knowledge,  comprehension,  application,  analysis,  synthesis,  and  evaluation. 
These  levels  are  identical  to  the  major  categories  in  the  cognitive  domain 
of  the  taxonomy  of  educational  objectives  described  in  Bloom,  1956.  We 
chose  to  call  them  cognitive  levels  instead  of  cognitive  categories  to 
establish,  implicitly,  a mental  picture  of  the  hierarchical  cognitive 
complexity  existing  among  different  cognitive  levels.  The  description  of 
the  six  cognitive  categories  have  been  summarized  in  Gronlund,  1970: 

" Knowledge  is  defined  as  the  remembering  of  previously  learned 
material.  This  may  involve  the  recall  of  a wide  range  of 
material,  from  specific  facts  to  complete  theories,  but  all 
that  is  required  is  the  bringing  to  mind  of  the  appropriate 
information.  Knowledge  represents  the  lowest  level  of 
learning  outcomes  in  the  cognitive  domains. 

"Comprehension  is  defined  as  the  ability  to  grasp  the  meaning 
of  material.  This  may  be  shown  by  translating  material  from 
one  form  to  another  (words  to  numbers) , by  interpreting  material 
(explaining  or  summarizing) , and  by  estimating  future  trends 
(predicting  consequences  or  effects).  These  learning  outcomes 
go  one  step  beyond  the  simple  remembering  of  material,  and 
represent  the  lowest  level  of  understanding . 

"Application  refers  to  the  ability  to  use  learned  material  in 
new  and  concrete  situations.  This  may  include  the  application 
of  such  things  as  rules,  methods,  concepts,  principles,  laws, 
and  theories.  Learning  outcomes  in  this  area  require  a higher 
level  of  understanding  than  those  under  comprehension. 

"Analysis  refers  to  the  ability  to  break  down  material  into 
its  component  parts  so  that  its  organizational  structure  may 
be  understood.  This  may  include  the  identification  of  the 
parts,  analysis  of  the  relationships  between  parts,  and 
recognition  of  the  organizational  principles  involved. 

Learning  outcomes  here  represent  a higher  intellectual  level 
than  comprehension  and  application  because  they  require  an 
understanding  of  both  the  content  and  the  structural  form 
of  the  material. 
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:|j*.  "Synthesis  refers  to  the  ability  to  put  parts  together  to  form 

a new  whole.  This  may  involve  the  production  of  a unique 
communication  (theme  or  speech),  a plan  of  operations  (research 
proposal) , or  a set  of  abstract  relations  (scheme  for  classifying 
information) . Learning  outcomes  in  this  area  stress  creative 
behaviors,  with  major  emphasis  on  the  formulation  of  new  patterns 
or  structures. 

"Evaluation  is  concerned  wi-th  the  ability  to  judge  the  value 
of  material  (statement,  novel,  poem,  research  report)  for  a given 
purpose.  The  judgments  are  to  be  based  on  definite  criteria. 

These  may  be  internal  criteria  (organization)  or  external 
criteria  (relevance  to  the  purpose)  and  the  student  may  determine 
the  criteria  or  be  given  them.  Learning  outcomes  in  this  area 
are  highest  in  the  cognitive  hierarchy  because  they  contain 
elements  of  all  of  the  other  categories , plus  conscious  value 
judgments  based  on  clearly  defined  criteria. " 

The  result  of  the  analysis  identified  the  highest  cognitive  level 
involved  in  each  behavior  as  well  as  the  degree  of  importance  of  that  cognitive 
level  for  creating  the  corresponding  behavior  in  a decision  making  context. 

The  analysis  was  based  on  the  decision  training  behavioral  objectives  developed 
previously  (Table  14).  The  result  of  this  analysis  is  summarized  in  Table  16. 
The  column  with  entries  identifies  the  highest  cognitive  level  associated 
with  the  corresponding  behavior.  The  entry  itself  represents  the  degree  of 
r importance  of  the  corresponding  cognitive  level/decision  behavior  pair. 

( 

In  order  to  assign  the  entries,  the  importance  of  each  of  the  subtasks 
involved  in  a decision  making  process  was  analyzed  and  three  different  levels 
of  importance  were  identifed.  An  entry  equal  to  1/2,  1 or  2 was  assigned  to 
the  subtasks  in  low,  middle  or  high  level  of  importance,  respectively.  Such 
assignment  of  entries  resulted  in  equal  values  for  the  summation  of  the  weights 
assigned  to  each  of  the  two  major  subtasks.  Since  problem  structuring  and 
alternative  selection  are  of  similar  importance  to  a decision  making  process, 
resulting  equal  values  can  be  considered  as  a confirming  evidence  to  the 
validity  of  this  method  of  assignment. 

Since  Type  1 decision  tasks  involve  only  alternative  formulation  and 
outcome  estimation,  the  summation  of  all  degrees  of  importance  associated 
with  each  cognitive  level  corresponding  to  behaviors  relevant  to  these  two 
decision  subtasks  provides  an  element  of  a weight  vector  for  cognitive  levels 
associated  with  Type  l decision  tasks.  Such  a summation  was  performed  for 
all  cognitive  levels  to  provide  the  complete  cognitive  level  weight  vector 
for  Type  1 decision  tasks.  A similar  procedure  was  performed  on  behaviors 
relevant  to  utility  assessment,  probability  assessment,  decision  rule  applica- 
tion, and  best  alternative  selection  subtasks  to  provide  cognitive  level 
weight  vector  for  Type  2 decision  tasks.  Since  Type  3 decision  tasks  involve 
the  subtasks  relevant  to  both  Type  1 and  Type  2 decisions,  the  cognitive  level 
weight  vector  for  Type  3 decision  tasks  was  resulted  by  simply  adding  the 
corresponding  weights  obtained  for  Type  1 and  Type  2 decision  tasks.  The 
weight  vectors  for  the  three  decision  types  are  shown  in  Table  17.  The  elements 
of  each  vector  have  been  normalized  such  that  the  summation  of  elements  adds  up 
to  one. 
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TABLE  16.  HIGHEST  COGNITIVE  LEVEL  IDENTIFICATION  AND  WEIGHT 
ASSIGNMENT  FOR  DECISION  MAKING  BEHAVIORS 


Cognitive 

Level 


Decision 

Behavior 


cn  i ba 


TABLE  17.  COGNITIVE  LEVEL  WEIGHT  MATRIX 
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A high  value  element  represents  a high  degree  of  importance  of  the 
corresponding  behavior  with  respect  to  the  decision  type.  Lower  value 
elements  define  a lesser  degree  of  importance  for  the  corresponding  behaviors. 
The  degrees  of  importance  of  the  lowest  cognitive  level  (knowledge)  and  the 
highest  cognitive  level  (evaluation)  for  Type  1 decision  tasks  are  very  low. 
These  degrees  can  be  approximated  by  zero  without  resulting  substantial 
inaccuracy.  The  cognitive  level  weights  for  the  three  decision  types  are 
plotted  in  Figures  18,  19,  and  20.  These  figures  indicate  that  the  resulted 
weights  correspond  to  intuitive  judgment.  The  occurrence  of  the  maximum 
degree  of  importance  for  Type  1 around  Analysis  level  is  justified  by  the 
fact  that  the  process  of  problem  structuring  mainly  involves  analysis  of  the 
problem  in  order  to  identify  the  relevant  elements  of  the  problem  as  well 
as  the  anlaysis  of  possible  outcomes  in  order  to  identify  their  effects  on 
the  problem  elements.  Since  Type  2 decision  tasks  mostly  involve  application 
of  certain  rules  and  procedures,  the  maximum  degree  of  importance  for  Type  2 
decisions  occurs  around  Application  level.  For  Type  3 decision  tasks  the 
degrees  of  importance  are  spread  along  different  cognitive  levels  with 
relatively  lower  variances.  In  general  the  process  of  decision  making 
mainly  concentrates  around  Analysis  (of  the  problem  and  potential  solution 
outcomes)  and  Application  (of  a set  of  rules  and  procedures). 

INSTRUCTIONAL  METHOD  SELECTION 

The  approach  to  solving  the  problem  of  instructional  method  selection  was 
to  establish  a quantification  scheme  which  can  be  used  to  rank  different 
instructional  methods  as  to  their  effectiveness  for  the  training  of  any 
decision  task.  To  implement  this  approach,  an  analysis  of  different  instruc- 
tional methods  was  performed.  Fourteen  major  instructional  methods  were 
selected  and  attributes  of  each  method  were  identified.  The  methods  include: 


(d)  Discussion  - Group 

(e)  Programmed  Instruction 

(f)  Games 

(g)  Simulation 

(h)  Projects  - Individual 

(i)  Projects  - Group 

(j)  Laboratory 

(k)  Apprenticeship 

(l)  Demonstration 
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Figure  19.  The  Degrees  Of  Importance  Of  Different 

Cognitive  Levels  For  Type  2 Decision  Tasks 
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Figure  20.  The  Degrees  Of  Importance  Of  Different 

Cognitive  Levels  For  Type  3 Decision  Tasks 
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(m)  Individual  Investigation  (Research) 

(n)  Text  - Reading 

Based  on  the  characteristics  of  each  instructional  method,  the  degree  of 
effectiveness  of  that  method  for  training  different  tasks  with  different 
levels  of  cognitive  complexity  were  judgmentally  evaluated.  The  judgmental 
criteria  were  of  two  fold:  (1)  the  effectiveness  of  the  method  judged  by 
the  expected  result  of  applying  the  method  in  a prototype  training  environ- 
ment and  (2)  the  effectiveness  of  the  method  judged  by  implicit  consideration 
of  cognitive  models  of  human  learning.  The  judgmental  data  related  to  the 
first  criterion  was  provided  by  a training  specialist,  and  the  data  concerning 
the  second  criterion  was  provided  by  a psychologist.  Lack  of  substantial 
variance  between  the  corresponding  judgmental  values  provided  by  the  two 
criteria  suggests  itself  as  confirming  evidence  for  the  correctness  of  the 
judgments.  However,  an  accurate  assessment  of  these  judgmental  data  demands 
a more  refined  procedure,  performed  in  a controlled  environment.  Due  to  the 
importance  of  such  data  for  research  projects  in  the  area  of  training 
cognitive  processes,  further  research  to  elicit  a more  accurate  assessment 
of  the  data  appears  to  be  needed. 

The  judgmental  data  resulting  from  the  two  criteria  were  averaged  to 
provide  an  aggregate  assessment.  The  expected  cost  and  time  of  instruction 
for  each  method  were  also  assessed  by  the  training  expert.  The  results  of 
the  assessment  is  shown  in  Table  18  as  Method/Attribute  Matrix.  The  entries 
of  the  matrix  have  value  ranges  from  one  to  ten.  The  higher  the  value,  the 
more  effective  the  method  for  training  a task  with  the  corresponding  level 
of  cognitive  complexity. 

This  information  suggests  a procedure  for  its  utilization  in  instructional 
method  selection.  Once  the  type  of  the  decision  task  to  be  taught  becomes 
known,  the  cognitive  level  weighting  vector  relevant  for  the  task  is  easily 
derived  from  the  cognitive  level  weight  matrix.  Then,  a linear  evaluation 
procedure  can  be  performed  by  a summation  over  the  results  of  multiplication 
of  the  elements  of  the  weight  vector  by  corresponding  elements  of  each  vector 
associated  with  different  methods.  That  is,  if  the  cognitive  level  weight 
vector  for  the  decision  type  associated  with  the  task  is  defined  as 
(w,,  ...  w . , ...  w,),(the  weights  for  "cost"  and  "time"  with  respect  to  the 
specific  training  environment  are  assigned  by  the  system  user  instructional 
technologist  during  system  application  as  (w^,  w„j),  and  the  vector  associated 
with  the  .th  method  is  defined  as  (m.,,  ...,  m..,  ...,  m.„),  then  the 
effectiveness  value  of  the  .th  i nstrtKtional  method  withJrespect  to  the 
decision  task  will  be:  J 


This  procedure  will  provide  an  effective  value  assessment  for  each  instruc- 
tional method.  Then  the  method  with  the  highest  effectiveness  value  will  be 
selected  as  the  most  effective  instructional  method  for  training  the  decision 
tasks  under  consideration. 


{ 
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TABLE  18.  METHOD/ATTRIBUTE  MATRIX 


Uri 11/Practice/Review 
Lecture 

Discussion-lnd.vidual  Tutorial 
Discussion-Group 
Programmed  Instruction 
Games 

Simulation 
Projects- 1 nd 1 vidua  1 
Projects  Group 
Laboratory 
Apprenticeshi 
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If  all  the  values  involved  in  the  procedure  for  selection  of  the  best 
instructional  method  were  invariant,  a method  could  be  selected  as  the  best 
instructional  method  for  each  decision  type.  Then,  identification  of  the 
type  of  a decision  task  would  directly  suggest  the  best  instructional  method 
for  the  decision  task.  However,  there  are  some  elements  whose  values  depend 
on  the  characteristics  of  the  training  environment  in  which  the  training 
program  will  be  implemented  ("Cost"  and  "Time",  in  this  case).  The  informa- 
tion concerning  the  characteristics  of  the  training  environment  will  be 
conveyed  by  the  system  user  instructional  technologist  during  system's 
application.  This  information  will  be  presented  in  the  form  of  judgmental 
importance  weights  assigned  to  a set  of  attributes.  The  set  of  attributes 
include:  Learner  paced  control  on  the  instructional  media,  instructional 
media  to  be  self  instructional,  motion  of  the  instructional  media,  fidelity 
of  the  instructional  media  in  terms  of  situational  details,  learner  control 
over  the  instructional  speed,  flexibility  of  the  media  for  instructional 
sequencing,  instruction  readability,  context  creation,  convenience  for 
development  of  testing  and  evaluation,  low  cost  of  instruction,  and  simplicity 
of  use  and  maintenance.  Furthermore,  introduction  of  the  selection  process 
as  a dynamic  procedure  will  provide  for  the  possibility  of  modification  of 
the  judgmental  values  as  well  as  addition  or  deletion  of  attributes. 

METHOD/MEDIA  RELEVANCY 

Analysis  of  different  instructional  media  identified  twelve  media  of 
major  interest: 

a.  Oral  Representation  (Lecture) 

b.  Textbook 

c.  Workbook/Problem  Sets 

d.  Pictures,  Slides,  Transparencies 

e.  Audio  Tape 

f.  Motion  Picture 

g.  TV/VTR 

h.  Programmed  Text 

i.  Computer  Assisted  Instruction  (CAI) 

j.  Part-Task  Trainer 

k.  Simulator 

l.  Natural  Object 

Since  some  of  these  instructional  media  are  not  applicable  to  a training 
environment  employing  a specific  instructional  method,  a study  of  relevancy 
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of  each  instructional  media  to  each  of  the  selected  instructional  methods 
was  performed.  The  result  of  the  study  dichotomized  the  relevant  and  irrele- 
vant media  for  each  instructional  method.  These  results  app  ar  in  the 
Method/Media  Relevancy  Matrix,  Table  19.  The  method/media  relevancy  matrix 
performs  as  a masking  scheme.  That  is,  each  zero  entry  rules  out  the 
acceptability  of  an  instructional  media  for  the  corresponding  instructional 
method.  A more  detailed  analysis  can  be  performed  to  evaluate  each  method/ 
media  pair  judgmental ly.  The  result  of  such  analysis  will  be  a degree  of 
relevancy  for  each  method/media  pair.  In  this  case  the  process  of  dichotomi- 
zation  of  media  for  each  method  will  be  replaced  by  a multilevel  evaluation 
process  which  will  provide  more  information.  For  example,  although  oral 
representation  and  textbook  are  both  considered  equally  relevant  to  drill/ 
practice/review  by  the  dichotomization  of  media,  a multilevel  evaluation 
might  also  convey  the  information  that  text  book  would  be  more  relevant  to 
drill/practice/review  by  assigning  numbers  on  a scale  of  1 to  10  as  the 
degree  of  relevancy  for  the  two  media  with  respect  to  the  method. 

INSTRUCTIONAL  MEDIA  SELECTION 

Once  the  best  instructional  method  for  a decision  task  has  been 
selected,  the  relevant  instructional  media  for  the  selected  method  can  be 
easily  identified  by  reference  to  the  method/media  relevancy  matrix.  The 
information  concerning  relevant  media  together  with  the  information  about 
the  characteristics  of  the  instructional  environment  provide  the  required 
data  for  selection  of  the  best  media. 

To  design  an  instructional  media  selection  methodology  based  on  this 
information,  an  analysis  of  the  characteristics,  requirements,  and 
resources  of  possible  instructional  environments  was  performed.  The 
result  of  the  analysis  identified  eleven  attributes  characterizing  different 
instructional  environments: 

a.  Learner  Paced  (control) 

b.  Self  Instructional 

c.  Motion/Still 

d.  Situational  Detail  (Fidelity) 

e.  Control  Over  Time  (Motion) 

f.  Flexible  Sequencing 

g.  Repeatability 

h.  Context  Creation 


i.  Potential  for  Test/Evaluation 
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TABLE  19.  METHOD/MEDIA  RELEVANCY  MATRIX 

I 


t 
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The  degree  of  effectiveness  of  each  instructional  media  with  respect  to 
different  attributes  was  judgmental ly  assessed  by  a training  specialist.  The 
criterion  for  assessment  in  this  case  was  the  effectiveness  of  the  media 
judged  by  the  expected  result  of  applying  the  media  in  a prototype  training 
environment  characterized  by  the  corresponding  attribute.  The  results  of 
this  assessment  appears  in  Table  20.  The  entries  of  the  matrix  have  value 
ranges  from  one  to  ten.  The  higher  the  value,  the  more  effective  the  media 
for  a training  environment  characterized  by  the  corresponding  attribute. 
Although  the  assessment  process  was  made  as  refined  as  possible  within  the 
time  limit  of  this  project,  a more  detailed  research  of  this  subject,  for 
eliciting  more  accurate  assessment  of  the  degrees  of  effectiveness,  seems 
advisable. 

The  Media/Attribute  Matrix  is  used  to  select  the  best  instructional  media 
for  a selected  instructional  method  to  be  used  in  a specific  instructional 
environment.  The  degrees  of  importance  of  attributes  with  respect  to  the 
training  environment  is  defined  as  a vector  presented  by  training  designer. 

This  vector  acts  as  a weight  vector  which  can  be  defined  as  (C,,  ...»  C., 

...,  C,,).  If  the  vector  associated  with  the  uth  media  in  medla/attribite 
matrix  is  defined  as  (a^j,  ...,  a^-,  ...»  a^-j),  and  the  vector  associated  with 
the  selected  method  in  Method/Media  relevancy  matrix  is  defined  as  (r,,  ..., 
r. , ...,  r , p ) ; then  the  degree  of  effectiveness  for  the  ,th  media  witn  respect 
to  the  selected  method  and  the  instructional  environmental' 1 1 be  calculated  as: 


The  expression  i Cj  a,,;  is  a simple  weighted  average  which  results  a number 
to  represent  an  overall  effectiveness  of  the  media  with  respect  to  the 
instructional  environment.  Depending  on  whether  the  media  is  compatible 
to  the  selected  instructional  method  or  not  the  value  of  ru  may  be  one  or 
zero;  this  means  that  the  value  of  E|<  will  be  equal  to  either  ^ C-j  a^j  or 
zero,  respectively.  Therefore,  the  result  of  applying  this  procedure  to  all 
twelve  media  will  be  twelve  numbers  each  representing  the  overall  effective- 
ness of  a media  based  on  both  the  characteristics  of  instructional  environ- 
ment and  the  selected  instructional  method.  The  method  with  the  highest 
overall  effectiveness  value  will  be  selected  as  the  most  effective  media  to 
be  used  with  the  selected  instructional  method  in  the  specific  instruction 
environment. 

PROCEDURE  FOR  INSTRUCTIONAL  GUIDELINE  GENERATION 

The  process  of  instructional  guideline  generation  for  decision  tasks  is 
compatible  with  conventional  Instructional  System  Development  (ISO).  This 
process  can  be  considered  as  a module,  within  a structured  ISD  program,  for 
recognition  and  classification  of  decision  tasks  as  well  as  selection  of  the 
most  effective  instructional  methods,  media,  and  contents  for  those  tasks. 
The  process  of  instructional  guideline  generation  can  be  defined  as  a syste- 
matic execution  of  a series  of  steps: 

a.  Define  Objective  Hierarchies  for  the  Training  Tasks.  As  in  the 
case  of  any  ISD  program,  the  process  of  instructional  guideline 
generation  starts  with  identification  of  objective  hierarchies 
for  the  training  tasks. 
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TABLE  20.  MEDIA/ATTRIBUTE  MATRIX 
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b.  Recognize  Decision  Tasks.  The  objective  hierarchies  are  processed 
by  the  decision  task  identi fication  scheme  to  recognize  the  decision 
tasks  involved  in  the  training  tasks. 

c-  Identify  Type  and  Class  of  the  Decision  Task.  Using  the  method 
described  as  decision  task  classification  preprocessor,  the  type 
and  the  class  of  each  decision  task  are  identified. 

d*  Select  the  Instructional  Contents . The  instructional  content  sections 
corresponding  to  different  attributes  of  the  class  of  decision  task 
are  combined  to  construct  the  basic  instructional  contents.  The 
basic  instruction  content  includes:  general  definition,  amplifi- 
cation, rule,  pitfalls  and  limitation,  interactions  with  other 
decision  elements,  and  prerequisites  for  each  instructional  content 
section.  In  addition  to  basic  instructional  content,  the  decision 
tree,  representing  each  decision  task,  is  constructed  to  provide 
the  required  task-specific  information  for  instructional  content. 

e.  Identify  Cognitive  Level  Weights.  The  cognitive  level  weights 
vector  associated  with  the  'decision  type  is  identified  by  a simple 
table  look-up  procedure. 

f.  Select  the  Most  Effective  Instructional  Method.  A linear  evaluation 
procedure  is  performed  by  a summation  over  the  results  of  multipli- 
cation of  elements  of  the  cognitive  levels  weight  vector  by 
corresponding  elements  of  each  vector  associated  with  different 
methods  in  the  method/attribute  relevancy  matrix.  This  procedure 
provides  an  effective  value  assessment  for  each  instructional 
method.  The  method  with  the  highest  effectiveness  value  will  be 
selected  as  the  most  effective  instructional  method  for  training 

the  decision  task  under  consideration. 

g . Identify  th e Instructional  Media  In compatible  with  the  Selected  Method . 
The  vector  associated  with  the  selected  instructional  method  in  the 
method/media  relevancy  matrix  is  identified  by  a simple  table 
look-up.  Those  instructional  media  corresponding  to  zero-value 
entries  in  the  vector  are  considered  as  the  ones  incompatible  with 

the  selected  method. 

h.  Select  the  Most  Effective  Instructional  Media.  The  degrees  of 
importance  of  attributes,  associated  with  each  media,  with  respect 
to  the  training  environment  is  defined  in  a weight  vector.  Then,  a 
linear  evaluation  procedure,  similar  to  the  one  used  in  item  (f), 
is  performed  to  select  the  most  effective  instructional  media.  A 
summation  is  performed  on  the  results  of  multiplication  of  elements 
of  the  weight  vector  by  corresponding  elements  of  each  vector 
associated  with  different  media  in  the  media/attribute  matrix.  The 
result  of  this  summation  will  define  the  degree  of  effectiveness 

of  the  media  if  the  media  is  compatible  with  the  selected  method. 

In  the  case  that  the  media  is  not  compatible  the  degree  of  effec- 
tiveness will  be  zero.  The  media  with  the  highest  degree  of 
effectiveness  will  be  selected  as  the  most  effective  instructional 
method  to  be  used  with  the  selected  instructional  method  in  the 
specific  instructional  environment. 
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SECTION  V 

CONCLUSIONS  AND  RECOMMENDATIONS 

GENERAL 

A system  to  generate  instructional  guidelines  for  a task-specific 
decision  training  program,  in  any  training  environment,  was  developed.  The 
system  selects  the  most  effective  instructional  method  and  media  and  provides 
required  instructional  contents  for  any  specific  decision  task,  subject  to 
training  in  a specific  training  environment.  The  project  has  been  aimed 
not  only  to  deve’op  a systematic  procedure  for  providing  instructional 
guidelines  for  decision  tasks  encountered  in  LAMPS  ASW  operation,  but  also 
to  design  a task-independent  methodology  in  order  to  make  application  of 
the  system  in  other  task  domains  subject  to  minimal  modification  effort. 

Such  a methodology  allows  incorporation  of  general  purpose  decision  training 
components  into  task  specific  decision  training  programs.  This  incorporation 
provides  trainees  with  the  basic  skills  required  for  making  task-specific 
decisions  within  a new  domain,  in  case  of  their  transfer  to  other  task  domains. 
Furthermore,  since  the  characteristics  of  decision  tasks  within  any  specific 
domain  are  subject  to  change  with  time,  the  methodology  will  allow  development 
of  more  effective  training  programs  even  in  the  case  where  there  is  no 
possibility  of  transferring  the  trainees  to  other  task  domains.  Finally,  the 
methodology  will  provide  for  the  carry-over  of  the  decision  making  capability 
into  incidental  functions,  which  are  usually  left  untrained. 

The  instructional  content  was  presented  in  two  categories:  task-specific 
information  and  basic  instructional  content.  The  task  specific  information 
is  provided  by  decision  trees  resulting  from  a detailed  analysis  of  the 
decision  trees  resulting  from  a detailed  analysis  of  the  decision  tasks.  The 
basic  instructional  content  is  generated  through  a classification  of  the 
decision  tasks  into  one  of  the  possible  classes  relevant  to  decision  training. 
Instructional  methods  and  media  are  selected  by  a quantified  scheme  which  takes 
into  account  the  characteristics  of  the  training  environment  as  well  as  the 
type  of  the  decision  tasks. 

The  Instructional  Guideline  Generation  System  provides  a systematic 
procedure  for  incorporating  a great  deal  of  data  into  the  process  of  selecting 
different  components  of  a decision  training  program.  These  data  are  of  great 
importance  to  any  instructional  system  development  and,  theoretically,  should 
all  be  considered  in  the  development  of  any  program.  The  procedure  of 
instructional  guideline  generation,  developed  in  this  project,  provides 
systematic  means  for  the  management  of  data  and  its  effective  utilization  in 
the  process  of  instructional  system  development.  However,  a more  effective 
utilization  of  the  instructional  guideline  generation  system  requires  a more 
detailed  look  into  three  problems:  (1)  elicitation  of  more  accurate  judgmental 
data,  (2)  a control  structure  to  guide  the  use  in  executing  different  steps 
of  the  procedure,  and  (3)  a computational  aid  for  calculations  required  in 
the  procedure. 

A computerized  instructional  guideline  generation  aid  will  provide  a 
vehicle  for  analysis  of  all  three  problems.  Such  an  aid  and  a plan  for 
realization  of  it  is  addressed  in  the  next  section. 
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COMPUTERIZED  INSTRUCTIONAL  GUIDELINE  GENERATION  AID 

Effectiveness  of  the  instructional  guideline  generation  system  can  be 
highly  enhanced  by  improvement  of  the  system  in  three  dimensions: 

a.  elicitation  of  more  accurate  judgmental  data 

b.  generation  of  a control  structure  to  guide  the  user  in 
executing  different  steps  of  the  system 

c.  development  of  a computational  aid  for  calculations  required 
in  the  system. 

A computerized  instructional  guideline  generation  aid  can  result  in  substantial 
improvements  in  all  three  dimensions. 

The  aiding  system  follows  the  design  described  in  Chapter  IV.  It  has 
the  potential  for  working  in  two  different  modes:  development  mode  and 
application  mode.  Development  mode  will  provide  access  to  judgmental  data 
stored  in  the  system.  The  accuracy  of  each  element  of  the  data  will  be 
assessed  by  an  analysis  of  the  guidelines  generated  for  decision  tasks  used 
as  test  cases.  The  inaccurate  data  may  be  modified  to  adapt  the  system's 
output  to  desirable  responses.  Application  mode  represents  a man/machine 
interactive  session  through  which  the  information  concerning  the  character- 
istics of  the  training  environment,  as  well  as  the  input  decision  task,  is 
elicited  from  the  user  upon  which  the  instruction  guidelines  for  the  decision 
task  will  be  generated.  During  interactive  sessions,  the  user  will  be 
guided  by  the  sequence  and  content  of  a series  of  questions  designed  to 
elicit  the  required  information.  Furthermore,  all  calculations  will  be 
performed  by  the  system  without  user's  interaction.  Such  a system  simplifies 
the  process  of  instructional  guideline  generation  into  a series  of  judgmental 
evaluations  of  elementary  attributes.  While  the  process  is  highly  simplified 
for  the  user,  the  system  automatically  incorporates  all  the  information 
acquired  from  the  user  into  the  process  which  results  in  recommendations  for 
the  most  effective  instructional  method  and  media  as  well  as  the  required 
instructional  contents. 

A prototype  application  mode  session  of  the  system  is  simulated  in  the 
following  pages.  The  user  responses  have  been  underlined. 

Describe  the  characteristics  of  the  instructional  environment  by 
assigning  an  important  weight  (between  zero  and  one)  to  each  of  the  following 
attributes. 

1.  Learner  paced  control  on  the  instructional  media. 

AT  ( 1 ) = J_ 

2.  Instructional  media  to  be  self  instructional. 

AT ( 2 ) = .8 

3.  Motion  of  the  instructional  media. 

AT ( 3 ) = J 


104 


NAVTRAEQU I PCEN  77-C-0005-1 


4.  Fidelity  of  the  instructional  media  in  terms  of  situational  details. 
AT(4)  = .8 

5.  Learner  control  over  the  instruction  speed  (motion). 

AT(5)  = .6 

6.  Flexibility  of  the  media  foi  instructional  sequencing. 

AT ( 6 ) = .4 

7.  Instruction  readability. 

AT( 7 ) = .3 

8.  Context  creation. 

AT (8)  = .8 

9.  Convenience  for  development  of  testing  and  evaluation. 

AT (9)  = .6 

10.  Low  cost  of  instruction. 

AT(10)  = .3 

11.  Simplicity  of  use  and  maintenance. 

AT (11)  = ^9 


Here  is  the  attribute  list  with  the  weights  you  assigned. 


1.  WEIGHT 

2.  WEIGHT 

3.  WEIGHT 

4.  WEIGHT 

5.  WEIGHT 

6.  WEIGHT 

7.  WEIGHT 

8.  WEIGHT 

9.  WEIGHT 

10.  WEIGHT 

11.  WEIGHT 


0.70:  Learner  paced  control  on  the  instructional  media. 

0.80:  Instructional  media  to  be  self  instructional. 

0.30:  Motion  of  the  instructional  media. 

0.80:  Fidelity  of  the  instructional  media  in  terms  of  situation. 
0.60:  Learner  control  over  the  instruction  speed  (motion). 

0.40:  Flexibility  of  the  media  for  instructional  sequencing. 
0.30:  Instruction  readability. 

0.80:  Context  creation. 

0.60:  Convenience  for  development  of  testing  and  evaluation. 
0.30:  Low  cost  of  instruction. 

0.90:  Simplicity  of  use  and  maintenance. 


Options  are 


Type 


1. 

Weights  are  satisfactory. 

q 

2. 

Change  weight  of  an  attribute. 

c 

3. 

Display  complete  list. 

d 

OPTION  = c 


10.  WEIGHT  = 0.30:  Low  cost  of  instruction. 


OPTION  = q 


i 
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Here  is  the  attribute  list  with  the  weights  you  assigned 

1. "  WEIGHT  = 0.70:  Learner  paced  control  on  the  instructional  media. 

2.  WEIGHT  = 0.80:  Instructional  media  to  be  self  instructional. 

3.  WEIGHT  = 0.30:  Motion  of  the  instructional  media. 

4.  WEIGHT  = 0.80:  Fidelity  of  the  instructional  media  in  terms  of  situation. 

5.  WEIGHT  = 0.60:  Learner  control  over  the  instruction  speed  (motion). 

6.  WEIGHT  = 0.40:  Flexibility  of  the  media  for  instructional  sequencing. 

7.  WEIGHT  = 0.30:  Instruction  readability. 

8.  WEIGHT  = 0.80:  Context  creation. 

9.  WEIGHT  = 0.60:  Convenience  for  development  of  testing  and  evaluation. 

10.  WEIGHT  = 0.50:  Low  cost  of  instruction. 

11.  WEIGHT  = 0.90:  Simplicity  of  use  and  maintenance. 

Throughout  this  consultation  session  the  above  importance  weights  will  be 
considered  to  represent  the  elements  of  the  instructional  environment.  To 
present  a new  environment  cancel  the  current  session  and  start  a new  one. 

Press  RETURN  when  ready  to  continue. 

Introduce  the  new  instructional  task  by  its  instructional  objective. 

Enter  description:  press  the  ESC  key  then  RETURN. 

C 1 assify  the  Contact  as  U.S.  Ship,  Enemy  Ship,  or  Natural  Object . 

Do  you  want  to  change  or  correct  the  description?  Type  yes/no.  No 

What  is  the  time  pressure  on  course  representation? 

Specify  it  by  a time  pressure  factor  between  zero  and  one. 

Factor  = .3 

Does  the  objective  of  the  task  include  selection  of  an  alternative  from  a 
set  of  alternatives?  Yes 

Is  the  set  of  alternatives  given?  Yes 

Is  there  a completely  specified  procedure  for  the  alternative  selection? 

No 


Once  the  alternatives  are  given,  is  there  any  critical  time  pressure  on  the 
operator  for  fast  alternative  selection?  Yes 

Which  of  the  following  decision  rules  is  the  most  suitable  one  for  the  task? 

1.  Subjective  Expected  Utility  (SEU) 

2.  Mini-max  loss  (MMX) 

3.  Mini-min  loss  (MMN) 

4.  Mini -max  regret  (MMR) 

5.  Lexicograph  (LEX) 

6.  Hurwicz  (HUR) 

7.  Satisficing  (SAT) 


Enter  a number  between  1 and  7? 
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INSTRUCTIONAL  GUIDELINES 

Instructional  objective:  classify  the  contact  as  U.S.  Ship,  Enemy  Ship,  or 
Natural  Object. 

The  task  to  be  taught  is  a decision  task. 

Decision  Type:  Type  2 
Decision  Class:  2-PMMX 

Instructional  Content  Must  Include  the  Following  Items: 

Collecting  subjective  utilities  under  time  pressures.  Translating  outcomes 
into  utilities.  Aggregating  utilities.  Relation  of  loss  and  utility 
concept  of  minimizing  maximum  loss.  Effect  of  time  pressure  on  quality. 
Concept  of  payoff  tables. 

The  Best  Three  Suggested  Instructional  Methods  are: 

1.  Discussion-Individual  Tutorial  for  which  the  best  three 
instructional  media  are: 

1 . Programmed  Text 

2.  CAI 

3.  Lecture  (oral  presentation/chalkboard) 

2.  Individual  Investigation  (Research)  for  which  the  best  three 
instructional  media  are: 

1.  Natural  Objects 

2.  Textbook 

3.  Workbook,  problem  sets 

3.  Projects  - Individual  for  which  the  best  three  instructional  media 
are: 

1.  Natural  objects 

2.  Textbook 

3.  Workbook,  problem  sets 
Press  RETURN  when  ready  to  continue. 

EVALUATION  PLAN 

Two  general  evaluation  areas  should  be  reviewed  in  assessing  the  impact 
of  the  decision  training  guidelines  described  earlier  in  this  report.  They 
are:  (1)  evaluation  of  the  impact  on  training  system  design,  development 
and  implementation,  and  (2)  evaluation  of  the  impact  on  trainees.  It  should 
be  noted  that  as  one  proceeds  from  the  initial  instructional  design  stages 
to  ultimate  field  performance  more  and  more  intervening  variables  enter  into 
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the  picture.  The  topics  identified  below  are  presented  with  full  recognition 
that  the  decision  training  guidelines  are  only  one  of  several  important 
variables  which  effect  the  overall  instructional  systems  process. 

a.  Desjgn.  The  design  stage  refers  to  the  initial  phase  of 
instructional  system  development  in  which  decision  tasks 
are  identified  from  among  the  full  set  of  tasks  which 
constitute  LAMPS  operator  functions.  At  this  stage  a general 
approach  to  subject  matter  and  instruction  is  defined. 

Relevant  evaluation  questions  at  this  stage  involve  the 
areas  of  accuracy,  complexity,  time/effort,  and  expertise. 

. Does  the  classification  scheme  accurately  identify  all 
tasks  which  are  in  fact  decision  tasks?  What  modifica- 
tions are  required  to  improve  the  accuracy  of  classification? 

Can  the  guidelines  be  applied  easily,  rapidly? 

. What  level  of  expertise  in  behavior  decision  theory  is 
required  to  apply  the  guidelines. 

. What  level  of  expertise  in  instructional  methods/media 
is  required  to  apply  the  guidelines. 

Is  the  information  provided  to  the  instructional  designed 
sufficient  to  develop  a general  approach  to  instruction 
which  integrates  all  the  decision  tasks  to  be  taught? 

b.  Development.  Once  decision  tasks  have  been  identified  and 
characterized  in  terms  of  relevant  decision  theoretic  and 
instructional  guidelines,  instructional  materials  are 
developed  and  tested.  Questions  of  interest  in  the 
development  stage  involve  the  completeness  and  robustness 
of  the  guidance  information  provided. 

. To  what  extent  do  the  supporting  conceptual  materials 
meet  the  developer's  needs  for  subject  matter  guidance? 
Conversely,  how  much  time/effort  is  required  to  supplement 
content  guidelines? 

. To  what  extent  do  the  guidelines  provide  direction  to  the 
developer  in  selection  of  an  instructional  strategy? 

. What  are  the  effects  of  environmental  factors  such  as 
instructional  setting  and  trainee  groups  on  the  utility  of 
the  content/instructional  guidelines?  How  robust  are  the 
guidelines  in  the  face  of  variations  in  trainee  groups, 
training  settings? 

. How  much  assistance  do  the  guidelines  provide  when  making 
revisions  to  draft  instructional  materials  subsequent  to 
testing? 


i 
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c.  Implementation.  This  phase  of  instructional  systems  development 
includes  the  installation  and  management  of  the  instructional 
system.  It  will  be  of  interest  at  this  stage  to  determine  what 
problems  exist  in  the  areas  of  logistics  and  support,  acceptance, 
instructor  skills,  management,  and  integration  with  related  (non- 
decision) LAMPS  training. 

. What  problems  of  acceptance  of  the  instructional  system 
occur  for  students,  instructors? 

. Are  special  skills  in  instructional  technique  or  expertise 
in  subject  matter  required  of  instructors? 

. Does  management  of  instruction  pose  special  problems 
of  data  handling  or  analysis? 

Is  decision  training  complementary  to  and  supportive 
of  the  content  of  related  (non-decision)  LAMPS  training? 

d.  Training  Efficacy  --  Didactic  Instruction.  At  this  stage  of  the 
instructional  systems  development  process,  full  scale  classroom 
instruction  occurs.  Major  evaluation  questions  at  this  point 
deal  with  the  level  of  attainment  of  training  objectives  by 
trainees,  and  the  associated  costs  and  time  required  for 
attainment  of  objectives. 

. Do  all  trainees  achieve  minimally  acceptable  levels  of 
knowledge  and  skill  as  defined  in  course  objectives? 

What  areas  of  weakness,  strength  can  be  identified  for 
individual  students?  For  the  trainee  group  as  a whole? 

. What  costs  and  how  much  time  are  associated  with  attainment 
of  individual  objectives?  With  completion  of  the  course  as 
a whole? 

e.  Training  Efficacy  --  Training  Missions.  In  this  stage,  the 
LAMPS  student  pilot  is  exposed  via  training  missions  to  a 
variety  of  in-flight  situations  some  of  which  may  test  his 
decision  skills  as  well  as  provide  an  opportunity  for  direct 
decision  task  instruction  by  a LAMPS  instructor.  The  evaluation 
topics  identified  in  the  previous  section  apply  here  to  any 
in-flight  decision  task  instruction  which  may  occur.  In 
addition,  the  amount  of  transfer  from  the  didactic  situation 

is  of  interest  to  determine. 

. What  benefits  (detriments)  to  decision  task  performance 
during  in-flight  training  can  be  identified  (for  students 
who  have  had  didactic  decision  training)? 

. Has  in-flight  training  been  enhanced  in  terms  of  time/costs 
required  as  a result  of  transfer  from  the  didactic  situation? 
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(f)  Post-Training  Performance.  Satisfactory  on-the-job  performance, 
subsequent  to  instruction,  represents  the  end-point  of  the 
instructional  system.  Questions  of  interest  here  relate  to  the 
impact  of  the  instructional  system  on  long  term  performance  of 
LAMPS  pilots. 

. What  evidence  exists  that  decision  training  has  improved 
mission  effectiveness  or  safety? 

. Can  general  improvements  (deficiencies)  in  decision 

performance  as  a result  of  decision  training  be  demonstrated 
or  are  the  benefits  (detriments)  restricted  to  specific 
tasks/situations  for  which  training  was  provided? 

. What  is  the  retention  level  of  decision  skills  three  months 
after  training?  After  one  year?  After  two  years? 

. What  major  content  deficiencies  in  pilot  decision  skills 
remain?  What  modifications  to  the  content  coverage  of 
formal  decision  training  seems  necessary? 

Acquisition  of  data  to  answer  all  of  the  evaluation  questions  identified 
above  will  involve  a series  of  evaluation  efforts,  some  formal  and  some 
informal.  Table  21  summarizes  the  principal  evaluation  questions  to  be 
answered  by  stage  together  with  an  identification  of  sources/methods  for 
acquiring  necessary  data.  Much  of  the  evaluation  information  identified  will 
be  of  use  in  revision,  either  of  the  decision  training  guidelines  or  of  the 
decision  training  itself.  That  is,  formative  use  of  the  evaluation  data  is 
more  likely  to  occur  and  will  probably  be  of  more  benefit  than  summative 
evaluation,  the  overall  assessment  of  the  value  of  the  guidelines  and/or 
training.  Summative  evaluation  will  be  difficult  to  carry  out  for  a number 
of  reasons,  two  of  which  are  particularly  important.  First,  training 
settings  typically  do  not  lend  themselves  to  a true  experimental  or  even  a 
quasi-experimental  analysis  of  instructional  impact.  Control  of  internal 
and  external  threats  to  validity  (e.g.,  mortality,  selection,  history, 
maturation,  testing,  reactive  effects,  testing-training  interaction,  etc.) 
require  experimental  designs  which  usually  depend  on  more  freedom  in  selection 
of  trainees  and  treatment  conditions  that  the  operational  training  setting  will 
permit.  Second,  the  time,  costs,  preserverance  and  cooperation  required  to 
obtain  representative  and  comprehensive  post-instructional  performance  data 
are  great  and,  as  a result,  less  useful  proxy  measures  such  as  performance 
at  time  of  exit/graduation  from  training  are  typically  accepted  as  substitutes. 

In  spite  of  these  considerations,  Table  21  presents  an  overview  of 
an  idealized  approach  to  evaluation  in  order  to  document  the  full  scope  of 
evaluation  analysis  which  could  be  brought  to  bear.  The  actual  evaluation 
effort  that  is  mounted  should  represent  a compromise  between  the  value  of 
the  information  sought  for  each  of  the  six  general  areas  identified  and  the 
cost  and  difficulty  of  obtaining  it. 
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TABLE  21.  OVERVIEW  APPROACH  FOR  EVALUATING 
DECISION  TRAINING  GUIDELINES 


STAGE 

EVALUATION  TOPIC 

SOURCE  OF  DATA/METHOD 

COMMENT/LIMITATIONS 

Design 

Accuracy  of 
classification 

Comparison  to  expert 
judgment 

Inter- judge  reliability 
should  be  demonstrated 

Ease  of 
guidelines 

Time,  costs  expended 
by  designer 

Expertise  required 
to  use  guidel ines 

Identification  of 
decision  tasks  or 
instructional 
methods  which  pose 
problems  for 
designer 

Dependent  on  skill 
of  course  designer 

Generality  of 
approach  afforded 
by  guidel ines 

Opinion  of  designer, 
developer 

Subjective 

Develop- 

ment 

Adequacy  of 
guidel ines 

Identification  of 
areas  in  which 
additional  subject 
matter  or  instruc- 
tional information 
is  requires;  include 
measures  of  time  spent 
supplementing  guide- 
lines by  developer 

Dependent  on  skill 
of  developer 

Robustness  in 
face  of  varying 
training  settings, 
trainee  groups 

Similar  to  above 

Subjective 

Assistance  in 
making  revisions 

Opinion  of  developer 

Subjective 
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TABLE  21.  OVERVIEW  APPROACH  FOR  EVALUATING 

DECISION  TRAINING  GUIDELINES  (CONTINUED) 


STAGE 

EVALUATION  TOPIC 

SOURCE  OF  DATA/METHOD 

COMMENT/LIMITATIONS 

Imple- 

menta- 

tion 

Coordination  with 
related  training 

Identification  of 
unusual  logistical 
problems  or  require- 
ments 

Dependent  on  training 
site  chosen 

Acceptance 

Opinions  of  students, 
i nstructors 

Subjective 

Instructor  skills 
required 

Opinions  of 
instructors 

Dependent  on  skills 
of  instructors  chosen; 
subjective 

Management  of 
i nstructor 

Special  data  handling/ 
analysis  requirements 

Integration  with 
related  training 

Opinions  of  students, 
instructors 

Subjective 

Didactic 

Training 

Students  achieve- 
ment levels 

Individual  and  group 
achievement  of  ob- 
jectives (by  objective 
and  for  course  as  a 
whole) 

Requires  well  defined, 
comprehensive  set  of 
objectives;  existence 
of  an  alternative 
treatment  group  com- 
parison is  unlikely; 
true-experimental 
design  may  be  hard 
to  implement  in  op- 
erational setting 

Costs , time 

Same  as  above 

Same  as  above;  should 
include  presentation 
costs  as  well  as 
trainee  costs 

Training 

Missions 

Transfer  from 
didactic  train- 
ing 

Performance  of 
students  on  decision 
tarks  encountered 
in-flight 

See  comments  under 
"DIDACTIC  TRAINING" 
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TABLE  21.  OVERVIEW  APPROACH  FOR  EVALUATING 

DECISION  TRAINING  GUIDELINES  (CONTINUED) 


STAGE 

EVALUATION  TOPIC 

SOURCE  OF  DATA/METHOD 

COMMENT/LIMITATIONS 

Reduced  costs, 
time  of  in-fl ight 
training 

Number  of  missions 
missions  required  to 
achieve  proficiency 

See  comments  under 
"DIDACTIC  TRAINING" 

Post- 

Training 

Perfor- 

mance 

Improved  mission 
effectiveness 

Critical  incidents, 
pilot  opinion, 
mission  logs 

Difficulty  of 
collecting  operational 
data  and  subjective 

1 imitations 

Generality  of 
effects 

Pilot  opinion 

See  above 

Retention  of 
decision  skills 

Test  trainees 
periodical ly 
after  graduation 

Follow-up  difficult 
to  implement 

Course  areas 
requiring 

Critical  incidents, 
pilot  opinion,  mission 
logs,  periodic  tests 
of  graduates 

Combination  of  infor- 
mation from  three 
categories  above 
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DECISION  AIDS 

Analysis  of  the  possible  roles  of  a human  operator  as  a decision  maker 
identified  two  different  roles:  an  operator  as  a decision  maker  and  an 
operator  as  a decision  executor.  Although  these  two  roles  may  be  played  by 
the  same  person,  it  is  also  possible  that  different  individuals  be  responsible 
for  each  of  the  two  roles.  In  this  case,  an  individual  makes  the  decision  by 
providing  the  decision  solutions  which  identifies  the  best  couse  of  action  to 
be  taken  under  each  possible  environmental  condition.  The  decision  solution 
is  then  communicated  to  the  second  person  who  acts  as  a decision  executor. 

At  any  decision  time,  the  decision  executor  detects  the  current  environmental 
condition  and  implements  the  decision  solution  corresponding  to  the  detected 
condition.  The  communication  media  by  which  the  information  concerning 
decision  solution  is  transmitted  from  the  decision  maker  to  the  decision 
executor  constitute  the  decision  aid  (Figure  21). 

Analysis  of  Pilot/ATO's  and  Sensor  Operator's  jobs  suggests  the  need 
for  decision  aids  in  critical  decision  situations.  In  many  critical  decision 
tasks  there  is  not  enough  time  for  a reasonable  decision  making;  furthermore, 
the  knowledge  of  the  criticality  of  the  situation  might  further  complicate 
any  on  the  job  decision  making.  Meanwhile,  the  operator  can  not  afford  any 
non-optimal  decisions.  A decision  aid,  in  such  a situation,  will  provide  the 
operator  with  optimal  solutions  which  have  been  provided  in  advance  and  under 
no  time  pressure. 

In  this  context  the  term  decision  aid  has  a very  broad  meaning.  Decision 
aids  can  range  from  pre-programmed  "look-up"  matrices  and  tables  to  computer 
driven  displays.  Some  examples  of  decision  aids  are  as  follows:  (1)  Non-computer 
oriented.  Tables  and  matrices,  blank  standardized  forms  to  be  f i 1 le<T  out 
that  structure  the  decision  and  its  data,  grease  pencils  and  plastic  status 
boards,  warning  lights  in  the  cockpit,  and  the  like,  (2)  Computer  oriented. 
Visual  displays  and  computer  graphics,  automatic  warnings,  CRT  displays, 
adaptive  decision  aiding  (e.g.,  ADDAM),  and  computerized  sequence  of  interaction 
decision  making,  information  entry,  aggregation  and  display,  and  so  on.  Each 
decision  aid  can  itself  be  characterized  in  terms  of  a number  of  attributes 
such  as 


. Maintainability 
. Cost 

. Completeness  of  aiding  (disaggregation) 

. Turn-around  time 

. Amount  of  manual  input  required 

. Complexity  of  the  information  display 

For  example,  a simple  cockpit  warning  light  provides  very  rapid  and  very 
aggregated  decision  aiding.  Either  the  light  is  on  or  it  is  off.  On  the 
other  hand,  an  interactive,  computer  based  system  that  requires  input  from 
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Figure  21  . The  Role  Of  Decision  Aids  In 
A Decision  Making  Process 
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a decision  executor  takes  time,  but  it  is  more  complete  in  its  display 
(and  also  more  complex  than  a single  light).  Prepackaged  check  lists  are 
mid-way  in  terms  of  turn  around  time,  complexity  and  input  information 
required.  In  summary,  each  decision  task  can  be  supported  with  a decision  aid 
and  each  decision  aid  can  be  developed  with  the  desired  aid  characteristics 
in  mind. 
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APPENDIX  A 

ANALYSIS  OF  OBJECTIVE  HIERARCHIES  FOR  PILOT/ATO 

Struc-  Decision 

ture  Page  Task  No.  Type  Keyword  Task  Area 


r 

2 

1.1.7 

Type 

2 

Classify,  specify 
the  degree 

Contact  Classification 

i ' 

47 

1.1. 7. 3 

Type  2 

Determine  Whether 

3 

1.1. 1.4 

Type 

2, 

Determine  'That, 

Equipment  to  Carry 

Type 

2 

Determine  What 

3 

1.1. 1.5 

Type 

1 

State 

Flying  Methods 

r 

3 

1.1. 1.6 

Type 

2 

Determine  whether 

Mission  safety 

u 

3 

1.1. 1.6.1 

Type 

1 

State  all 

3 

1.1. 1.6 

Type 

2 

Determine  whether 

Mission  safety 

L 

3 

1.1. 1.6.4 

Type 

2 

Determine  whether 

3 

1.1. 1.7.1 

Type 

1 

State 

Crew  Briefing 

r 

14 

1.1. 4. 3.1 

Type 

3 

Recognize 

Malfunction  Recognition 

1- 

16 

1.1. 4. 3. 1.1 

Type 

3 

Recognize 

* 

16 

1.1. 4. 3. 1.1. 

■ Type 

2 

Indicate  which 

, 

1 

16 

1.1. 4. 3. 1.1. 

. 1 

Type 

1 

Describe  the 
components 

1 , 

17 

1.1. 4. 3. 1.1.1 

.1.2 

Type 

2 

State  the  1 i kely 
effect 

1 

14 

1.1. 4. 3. 2 

Type 

3 

Recognize 

Malfunction  Recognition 

i. 

18 

1.1. 4. 3. 2.1. 

Type 

2 

Indicate  which 

1 — 

14 

1.1. 4. 3. 3 

Type 

3 

Recognize 

Malfunction  Recognition 

1 „ 

19 

1.1. 4. 3. 3. 1.1 

Type 

2 

Indicate  which 

r 

14 

1.1. 4. 3. 4 

Type 

3 

Recognize 

Malfunction  Recognition 

I, 

20 

1.1. 4. 3. 4. 1.1 

Type 

2 

Indicate  which 

r~ 

14 

1 . 1 .4.3.5 

Type 

3 

Recognize 

Malfunction  Recognition 

u 

21 

1.1. 4. 3. 5. 1.1 

Type 

2 

Indicate  which 

14 

1.1. 4. 3. 6 

Type 

3 

Recognize 

Malfunction  Recognition 

1 

22 

1.1. 4. 3. 6. 1.1 

Type 

2 

Indicate  Which 

r~ 

14 

1.1. 4. 3. 7 

Type 

3 

Recognize 

Malfunction  Recognition 

U 

23 

1.1. 4. 3. 7. 1.1 

Type  2 

Indicate  which 

t 

14 

1.1. 4. 3. 8 

Type 

3 

Recognize 

Malfunction  Recognition 

24 

1.1. 4. 3. 8. 1.1 

Type  2 

Indicate  which 

c 

14 

1.1. 4. 3. 9 

Type 

3 

Recognize 

Malfunction  Recognition 

25 

1.1. 4. 3. 9. 1.1 

Type 

2 

Indicate  which 

i 

’• 
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Struc- 

Decision 

ture 

Page 

Task  No. 

Type 

Keyword 

14 

1.1.4.3.10 

T ype  2 

State,  most  likely 

27 

1.1.4.3.10.1 

Type  1 

State 

p 

14 

1.1.4.3.10 

LType  2 

State,  most  likely 

27 

1.1.4.3.10.3 

Type  1 

State 

— 

14 

1.1.4.3.10 

TType  2 

State  most  likely 

u 

27 

1.1.4.3.10.5 

LType  2 

State  the  most  likely 

*■* 

23 

1.1.4.3.10.5.2 

iType  2 

State  the  likely 
effect 

29 

1.1.4.3.11.1.2 

Type  2, 

Determine  the 

Type  2 

urgency,  if 

29 

1.1.4.3.11.3 

Type  3 

Recognize 

3) 

1.1.4.3.12.1.1 

Type  2 

Indicate  which 

32 

1.1.1  3.12.3.2 

LType  2 

State  1 ikely  effect 

33 

1.1.4.3.13.1.1 

Type  2 

Indicate  which 

34 

1.1.4.3.14.1.1 

Type  2 

Indicate  which 

3S 

1.1.4.3.15.1.1 

Type  2 

Indicate  which 

38 

1.1. 6.1 

Type  3 

Choose  an  appro- 
priate 

38 

1.1. 6. 1.1 

Type  2 

Determine  the  likely 
identity 

* 

39 

1 . 1 .6. 1 . 1 .5 

Type  2 

Determine  the  likely 
identity 

38 

1.1.6  1 

Type  3 

Choose  an  appro- 
priate 

u 

38 

1.1. 6. 1.2 

Type  1 

Determine  possible 
behavior 

i . 

40 

1.1. 6. 1.2.1 

Type  3 

Determine  the  most 
likely 

42 

1.1. 6. 1.2. 2. 3.  1.2. ' 

i.4  Type  2 

Classify 

- 

38 

1.1. 6.1 

Type  3 

Choose  an  appro- 
priate 

*■ 

38 

1. 1.6. 1.3 

Type  1 

State 

-* 

45 

1.1. 6. 1.3. 3 

Type  2 

Determine  the  pro- 

portion 


Task  Area 
Type  of  Fire 

Type  of  Fire 

Type  of  Fire 

Mission  Urgency 

Electric  Failure 
Recognition 

Malfunction  Recognition 

Fuel  System  Fai lure 

Tire  Blow-out 

Tire  Blow-out 

Loss  of  Tail  Rotor 
Control 

ASW  Mission  Tactic 


ASW  Mission  Tactic 


Sound/Veloci ty  Profile 
Interpretation 

ASW  Mission  Tactic 
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Struc- 

ture 


r. 

c 


E 

E 


Decision 


Page 

Task  No. 

Type 

Keyword 

38 

1.1. 6.1 

Type  3 

Choose  an  appro- 
priate 

38 

1.1. 6. 1.3 

Type  1 

State 

45 

1.1. 6. 1.3. 4 

Type  2 

Determine  the  depth 

38 

1.1. 6.1 

Type  3 

Choose  an  appro- 
priate 

38 

1.1. 6. 1.3 

Type  1 

State 

45 

1.1. 6. 1.3. 5 

Type  2 

Determine  if 

38 

1.1. 6. 2 

Type  3 

Revise  plan 

38 

1.1. 6. 2.1 

Type  2 

Determine  whether 

38 

1.1. 6. 2 

Type  3 

Revise  plan 

38 

1.1. 6. 2. 2 

Type  3 

Select  an  appro- 
priate 

48 

1.1. 7. 1.1. 3 

Type  2 

Determine  if 

48 

1.1. 7. 1.1. 4 

Type  3 

Prescribe  an 
appropriate 

48 

1.1. 7. 1.1. 5 

Type  2 

Determine  which 

49 

1.1. 7. 1.2. 4 

Type  3 

Determine  an 
appropriate 

51 

1.1. 7. 2. 2 

Type  3 

Select  an  appro- 
priate 

53 

1.1. 9. 6.1 

Type  2 

State  the  proper 

53 

1.1. 9. 6. 2 

Type  2 

State  the  proper 

55 

1.1.10.1.1 

Type  3 

Identi fy 

56 

1.1.10.1.1.1 

Type  3 

Recognize 

57 

1.1.10.1.1.1.1.1 

LType  2 

Indicate  which 
accompany 

55 

1.1.10.1.1 

Type  3 

Identify 

56 

1.1.10.1.1.2 

Type  3 

Recognize 

58 

1.1.10.1.1.2.1.1 

£Type  2 

Indicate  which 

55 

1.1.10.1.1 

Type  3 

Identify 

56 

1.1.10.1.1.3 

Type  3 

Recognize 

59 

1.1.10.1.1.3.1.1 

ZType  2 

Indicate  which 

55 

1.1.10.1.1 

Type  3 

Identify 

56 

1.1.10.1.1.4 

Type  3 

Recognize 

60 

1.1.10.1.1.4.1.1 

EType  2 

Indicate  which 

* 


Task  Area 

ASE  Mission  Tactic 

ASW  Mission  Tactic 

Tactic  Revision 

Tactic  Revision 

Pattern  Expansion 
Pattern  Expansion 

Sonobuoys  Monitoring 
Pattern  Expansion 

MAD  Tracking  Pattern 

Refuel ing 
Refuel ing 

Engine  Malfunction 

Engine  Malfunction 

Engine  Malfunction 

Engine  Malfunction 
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Struc- 

Decision 

ture 

Page 

Task  No. 

Type 

Keyword 

r 

55 

1.1. 10. l.l 

Type  3 

Identify 

F 

56 

1.1.10.1.1.5 

Type  3 

Recognize 

u 

61 

1.1.10.1.1.5.1.1 

EType  2 

Indicate  which 

r 

55 

1.1.10.1.1 

Type  3 

Identify 

F 

56 

1.1.10.1.1.6 

Type  3 

Recognize 

L* 

62 

1.1.10.1.1.6.1.1 

EType  2 

Indicate  which 

r 

55 

1.1.10.1.1 

Type  3 

Identify 

r 

56 

1.1.10.1.1.7 

Type  3 

Recognize 

I# 

63 

1.1.10.1.1.7.1.1 

EType  2 

Indicate  which 

64 

1.1.10.1.1.8.2 

EType  2 

State  the  likely 

effects 

- 

55 

1.1.10.1.2 

Type  3 

Identi fy 

- 

65 

1.1.10.1.2.1 

Type  3 

Recognize 

' 

66 

1.1.10.1.2.1.1.1 

EType  2 

Indicate  which 

55 

1.1.10.1.2 

Type  3 

Identify 

1* 

65 

1.1.10.1.2.2 

Type  3 

Recognize 

' * 

67 

1.1.10.1.2.2.1.1 

EType  2 

Indicate  which 

- 

55 

1.1.10.1.2 

Type  3 

Identify 

•• 

65 

1.1.10.1.2.3 

Type  3 

Recognize 

68 

1. 1.10. 1.2. 3. M 

EType  2 

Indicate  which 

69 

1.1.10.1.2.4.2 

EType  2 

State  the  1 i kely 

effect 

r 

55 

1.1.10.1.3 

Type  3 

Identi fy 

r 

70 

1.1.10.1.3.1 

Type  3 

Recognize 

* . 

71 

1.1.10.1.3.1.1.1 

EType  2 

Indicate  which 

| 

55 

1.1.10.1.3 

Type  3 

Identi fy 

► 

70 

1.1.10.1.3.2 

Type  3 

Recognize 

72 

1.1.10.1.3.2.1.1 

EType  2 

Indicate  which 

r 

55 

1.1.10.1.3 

Type  3 

Identi fy 

i* 

70 

1.1.10.1.3.3 

Type  3 

Recognize 

u 

73 

1.1.10.1.3.3.1.1 

EType  2 

Indicate  which 
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Task  Area 

Engine  Malfunction 


Engine  Malfunction 


Engine  Malfunction 


Engine  Malfunction 


Drive  System 
Mai  function 


Drive  System 
Malfunction 


Drive  System 
Malfunction 


Transmission 

Malfunction 

Electrical  System 
Malfunction 


Electrical  System 
Mai  function 


Electrical  System 
Malfunction 
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Struc- 

Decision 

ture 

Page 

Task  No. 

Type 

Keyword 

Task  Area 

r 

55 

1.1.10.1.3 

Type  3 

Identi fy 

Electrical  System 
Malfunction 

r 

70 

1.1.10.1.3.4 

Type  3 

Recognize 

u 

74 

1.1.10.1.3.4.1.1 

iType  2 

Indicate  which 

r 

55 

1.1.10.1.4 

Type  3 

Identify 

Fuel  System 

Malfunction 

75 

1.1.10.1.4.1 

Type  3 

Recognize 

L* 

76 

1.1.10.1.4.1.1.1 

tType  2 

Indicate  which 

r 

55 

1.1.10.1.4 

Type  3 

Identify 

Fuel  System 

Malfunction 

h 

75 

1.1.10.1.4.2 

Type  3 

Recognize 

u 

77 

1.1.10.1.4.2.1.1 

FType  2 

Indicate  which 

P 

55 

1.1.10.1.4 

Type  3 

Identify 

Fuel  System 

Malfunction 

f* 

75 

1.1.10.1.4.3 

Type  3 

Recognize 

u. 

78 

1.1.10.1.4.3.1.1 

/.Type  2 

Indicate  which 

79 

1.1.10.1.4.4.2 

iType  2 

State  the  likely 

Engine  Oil  System 

effects 

Malfunction 

80 

1.1. 10. 1 .4.5.2 

tType  2 

State  the  likely 

Transmission  Oil  System 

effect 

Malfunction 

r- 

55 

1.1.10.1.5 

Type  3 

Identify 

Oil  System  Malfunction 

81 

1.1.10.1.5.1 

Type  3 

Recognize 

u 

82 

1.1.10.1.5.1.1.1 

i.Type  2 

Indicate  which 

p 

55 

1.1.10.1.5 

Type  3 

Identify 

Oil  System  Malfunction 

81 

1.1.10.1.5.2 

Type  3 

Recognize 

83 

1.1.10.1.5.2.1.1 

zType  2 

Indicate  which 

84 

1.1.10.1.5.3.2 

TType  2 

State  the  1 ikely 
effect 

Main  Rotor  Malfunction 

85 

1.1.10.1.5.4.2 

t.Type  2 

State  the  likely 

Brake  System 

effect 

Mai  function 

86 

1.1.10.1.5.5.2 

tType  2 

State  the  likely 

Blade  Track  System 

effect 

Mai  function 

f — 

55 

1.1.10.1.6 

Type  3 

Identify 

Flight  Controls 
Malfunction 

j ► 

87 

1.1.10.1.6.1 

Type  3 

Recognize 

■ * 

88 

1.1.10.1.6.1.1.1 

tType  2 

Indicate  which 

89 

1.1.10.1.6.2.2 

/.Type  2 

State  the  likely 

Hydraulic  Power  Supply 

effect 

Malfunction 

125 


NAVTRAEQUIPCEN  77-C-0005-1 


Struc- 

Decision 

ture 

Page 

Task  No. 

Type 

Keyword 

r- 

55 

1.1.10.1.7 

Type  3 

Identify 

-*■ 

90 

1.1.10.1.7.1 

iiType  2 

Identify  those 

92 

1.1.10.2.1.1 

Type  2 

Identify  whether 

93 

1.1.10.2.2.3 

Type  2 

Determine  the  type 
(known  possibility 

95 

1.1.11.1.1.3 

Type  2 

Determine  whether 

97 

1.1.11.1.2.1.3 

Type  2 

Determine  it 

98 

1.1.11.1.2.2.3 

Type  2 

Determine  if 

99 

1.1.11.1.2.3.3 

Type  2 

Determine  if 

100 

1.1.11.1.2.4.3 

Type  2 

Determine  if 

101 

1.1.11.1.2.5.3 

Type  2 

Determine  if 

102 

1.1.11.1.3.1 

Type  2 

Determine  whether 

104 

1.1.11.1.3.2.3.2 

Type  2 

Determine  if 

106 

1.1.11.2.1.3 

Type  2 

Determine  whether 

107 

1.1.11.2.2.5 

Type  2 

Determine  the  min. 

107 

1.1.11.2.2.6 

Type  2 

Determine  whether 

u 

107 

1.1.11.2.2.6.1 

Type  3 

State  the  criteria 

r- 

107 

1.1.11.2.2.7 

Type  3 

Determine  which 

w 

107 

1.1.11.2.2.7.1 

Type  1 

State  the  alter- 
native 

108 

1.1.11.2.2.1.3 

Type  2 

Determine  if 

109 

1.1.11.2.2.2.3 

Type  2 

Determine  if 

110 

1. 1.11.2.2.3.6 

Type  2 

Determine  if 

111 

1.1.11.2.2.4.3 

Type  2 

Determine  when 

111 

1.1.11.2.2.4.5 

Type  2 

Determine  if 

112 

1.1.11.2.3.1 

Type  2 

Determine  whether 

112 

1.1.11.2.3.2.1 

LType  2 

State  the  cues 

Task  Area 

Hydraulic  System 
Malfunction 

Landing  Methods 

Enqine  Failure 

) ' 

Landing  Feasibi 1 i ty 
Land  or  Abort 
Land  or  Abort 
Land  or  Abort 
Land  or  Abort 
Land  or  Abort 
Land  or  Abort 
Enqine  Failure 
Landing  Feasibi 1 ity 
I lying  Method 
Land  or  Wave  Off 

Landing  Method 

Landing  Safety 
Landing  Safety 
Landing  Safety 
Flying  Method 
Landing  Safety 
Land  or  Abort 
Loss  of  Tail  Control 
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Struc- 

Decision 

ture 

Page 

Task  No. 

Type 

Keyword 

Task  Area 

114 

1.1.12.1.3.2 

Type  2 

Determine  if 

Postflight  Inspection 

115 

1.1.12.1.3.6.2 

Type  2 

Determine  if 

Postflight  Inspection 

116 

1.1.12.1.3.4.1.2 

Type  3 

Identify 

Tail  Wheel  Lock  Failure 

116 

1.1.12.1.3.4.2.1 

Type  3 

Recognize 

Total  Electrical 

Fa i 1 ure 

116 

1.1.12.1.3.4.3.1 

Type  3 

Recognize 

Droop  Stop  Failure 

116 

1.1.12.1.3.4.4.1 

Type  3 

Recognize 

Internal  Engine  Fire 

116 

1.1.12.1.3.4.5.1 

Type  3 

Identify 

External  Engine  Fire 

118 

1.1.13.2 

KType  3 

State 

Intelligence  Gathering 

118 

1.1.13.3 

xType  3 

State 

Tactics 

118 

1.1.13.4 

):Type  3 

State 

Crew  Briefing 

120 

1.2. 6. 3 

Type  2 

Determine 

Tactics 

120 

1 .2.6.4 

Type  3 

Determine  the 
appropriate 

Tactics 

■ 

120 

1.2. 6. 5 

Type  3, 

Determine, 

Tactics 

Type  3 

determine 

• 

120 

1.2. 6. 5.1 

Type  3, 

Determi ne , 

Type  3, 

determi ne , 

Type  3 

determi ne 

r 

120 

1 .2.6.5 

Type  3, 

Determine , 

Tactics 

Type  3 

determine 

L. 

120 

1 .2. 6. 5. 2 

Type  3, 

Determine, 

Type  3 

determine 

120 

1.2. 6. 6 

J.Type  3 

State  the  appro- 
priate ref's 

Tactics 

120 

1.2. 6. 7 

Type  3 

State 

Tactics 

123 

1.3. 5. 4. 2 

Type  3 

Classify 

Tactics 

123 

1.3. 5. 4. 3 

Type  3 

Classify 

Tactics 

123 

1.3. 5. 4. 4 

Type  3 

Classify 

Tactics 

■ 

128 

1.4. 6. 1.1. 2 

Type  3 

Determine  which 

Search  and  Rescue 

’ 

128 

1.4. 6. 1.1. 2.1 

Type  1 

State 

J 
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Task  No. 

Type 

Keywords 

Task  Area 

128 

1.4. 6. 1.1. 3 

Type  2 

Determine  the  best 

Search  and  Rescue 

128 

1.4. 6. 1.1. 4 

Type  3 

State 

Search  and  Rescue 

129 

1 .4.6.1 .3.1 .1 

Type  3 

Identify 

Search  and  Rescue 

129 

1.4. 6. 1.3. 3. 1.1 

Type  3 

State  the  cues 

Search  and  Rescue 

131 

131 

1.4. 6. 2. 1.2 

1.4. 6. 2. 1.2.1 

Type  3 
Type  1 

Determine  which 
State  the  major 

Search  and  Rescue 

132 

1 .4. 6. 2. 2. 2 

Type  3 

Determine  which 

Flying  Method 

132 

1 .4. 6. 2. 2. 3 

Type  1 

State  the  major 

Flying  Method 

133 

1 .4.7.1 .2.2 

Type  2 

Determine 

Landing  Feasibility 

134 

1 .4. 7. 2. 3. 1 

Type  2 

Determine 

Search  and  Rescue 

135 

1.4. 7. 3. 1.1. 1.2 

Type  3 

Locate 

Search  and  Rescue 

136 

1.4. 7. 5. 1.1 

Type  3 

Identify 

Doppler  Failure 

136 

1.4. 7. 5. 2.1 

Type  3 

Identify 

Radar  Altimeter  Failure 

139 

1.5. 5. 1.6 

Type  2 

State  al 1 appro- 
priate 

External  Cargo 

140 

1.5. 5. 3. 3 

Type  2 

State  all  appro- 
priate 

External  Cargo 

142 

1.6. 1.1 

Type  3 

Determi ne 

Internal  Cargo 

142 

1.6. 1.2 

Type  3 

State 

Internal  Cargo 

145 

1 .7.5.5 

Type  3 

Determi  nr 

Gunfire  Spotting 

150 

1 .8. 2. 2. 4 

Type  2 

State  likely 
effects 

l ighting  System 

T ai  lure 

152 

1.8. 2. 3. 1.2 

LType  2 

State  likely 
effect 

Flight  Instrument 

System  Failure 

153 

1.8. 2. 3. 2 2 

LType  2 

State  likely 
effect 

AN/AYK-2  System 

Fai lure 
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155 

1.8. 2. 3. 4. 2 

EType  2 

State  likely 
effects 

Standby  Compass 
Failure 

159 

1 .8.2. 3.8.2 

EType  2 

State  likely 
effects 

AN/APN-182  System 
Failure 

160 

1.8. 2. 3. 9. 2 

EType  2 

State  likely 
effects 

AN/ASN-50  System 
Failure 

163 

1.8. 2. 4. 1.4 

EType  2 

State  likely 
effects 

Sonobuoy  Processing 
System  Failure 

164 

1.8. 2. 4. 2. 2 

EType  2 

State  likely 
effects 

Smoke  Maker  Dispenser 
System  Failure 

165 

1.8. 2. 4. 3. 2 

EType  2 

State  likely 
effects 

Torpedo  Resease 

System  Failure 

166 

1.8. 2. 4. 6. 2 

EType  2 

State  likely 
effects 

Sonobuoy  Launcher 
System  Failure 

167 

1.8. 2. 4. 7. 2 

EType  2 

State  likely 
effects 

MAD  System  Failure 

168 

1.8. 2. 4. 8. 2 

EType  2 

State  likely 
effects 

ALR-54  System  Failure 

171 

1.8. 2. 5. 2. 2 

EType  2 

State  likely 
effects 

AN/ARC-159  System 
Failure 

174 

1.8. 2. 5. 5. 2 

EType  2 

State  likely 
effects 

Jul iet  28  System 

Fai lure 

182 

1 .8. 5. 3. 2 

EType  2 

State  likely 
effects 

Anti-Ice  System 
Failure 
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Struc- 

Decision 

ture 

Page 

Task  No. 

Type 

Keyword 

Task  Area 

r- 

3 

2. 1.1. 2 

Type  2 

Determine 

if 

Daily  Inspection 

5 

2. 1.1. 2.1 

Type  2 

Determine 

if 

c 

3 

2 . 1 . 1 . 2 

Type  2 

Determi ne 

if 

Daily  Inspection 

5 

2. 1.1. 2. 2 

Type  2 

Determine 

if 

p 

3 

2. 1.1. 2 

Type  2 

Determine 

if 

Daily  Inspection 

5 

2. 1 . 1 .2.3 

Type  2 

Determine 

if 

c 

3 

2. 1.1. 2 

Type  2 

Determine 

if 

Daily  Inspection 

5 

2.1. 1.2.4 

Type  2 

Determi ne 

if 

3 

2. 1.1. 2 

Type  2 

Determine 

if 

Daily  Inspection 

l. 

5 

2. 1.1. 2. 5 

Type  2 

Determi ne 

if 

c 

3 

2. 1.1. 2 

Type  2 

Determine 

if 

Daily  Inspection 

5 

2.1. 1.2.6 

Type  2 

Determi ne 

if 

3 

2. 1.1.2 

Type  2 

Determi ne 

if 

Daily  Inspection 

1 » 

5 

2.1. 1.2.7 

Type  2 

Determine 

if 

r. 

3 

2.1. 1.2 

Type  2 

Determine 

if 

Daily  Inspection 

5 

2. 1 . 1 .2.8 

Type  2 

Determine 

if 

3 

2 . 1 . 1 . 6 

Type  2 

Determi ne 

if 

Fuel  Acceptability 

r 

6 

2. 1 .2.2 

Type  2 

Determi ne 

if 

Turnaround  Inspection 

3 

2. 1.2. 2.1 

Type  2 

Determine 

if 

1 

6 

2. 1.2.2 

Type  2 

Determine 

i f 

Turnaround  Inspection 

i . 

8 

2. 1.2. 2. 2 

Type  2 

Determine 

if 

r 

6 

2. 1.2.2 

Type  2 

Determi ne 

if 

Turnaround  Inspection 

8 

2. 1.2. 2. 3 

Type  2 

Determine 

if 

r 

6 

2.  1.2.2 

Type  2 

Determine 

if 

Turnaround  Inspection 

8 

2. 1.2. 2. 4 

Type  2 

Determi ne 

if 

i 

6 

2. 1.2. 2 

Type  2 

Determine 

if 

Turnaround  Inspection 

1 » 

8 

2. 1 .2.2.5 

Type  2 

Determi ne 

if 

r— 

6 

2. 1.2.2 

Type  2 

Determine 

if 

Turnaround  Inspection 

1 * 

8 

2. 1.2. 2. 6 

Type  2 

Determine 

if 

C 

6 

2. 1.2. 2 

Type  2 

Determine 

if 

Turnaround  Inspection 

8 

2. 1.2. 2. 7 

Type  2 

Determine 

if 

6 

2.1.2  4 

Type  3 

Identify 

Standard  Safety 
Precaution 
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Page 

Task  No. 

Type 

Keyword 

Task  Area 

6 

2. 1.2. 6 

Type  2 

Determine  if 

Turnaround  Inspection 

1 

13 

2. 2. 1.7 

Type  2 

Cl  ass i fy 

Tactics,  Target 

! 

Classification 

-> 

41 

2. 2. 1.7.1 

Type  2 

Perform  al 1 classi- 

fication functions 

to  identify 

► 

42 

2. 2. 1.7. 1.1 

Type  2 

Classify 

r 

13 

2.2.1 .7 

Type  2 

Classify 

Target  Classification 

41 

2. 2. 1.7.1 

Type  2 

Perform  al 1 classi- 

fication functions 

to  identify 

L 

42 

2. 2. 1.7. 1.2 

Type  3 

Identi fy 

r 

13 

2. 2. 1.7 

Type  2 

Classify 

Target  Classification 

U 

41 

2. 2. 1.7.1 

Type  2 

Perform  all  classi - 

fication  functions 
to  identify 


..  42 

2. 2. 1.7. 1.3 

Type 

3 

Identi fy 

1 1? 

2. 2. 1.7 

Type 

2 

Classify 

Target  Classification 

* 41 

2. 2. 1.7.1 

Type 

2 

Perform  all  classi- 
fication functions 
to  identify 

. 42 

2. 2. 1.7. 1.4 

Type 

3 

Identi fy 

13 

2.2.1 .7 

Type 

2 

Classify 

Target  Classification 

f 

2. 2. 1.7.1 

Type 

2 

Perform  al 1 classi- 
fication functions 
to  identify 

1.  42 

2. 2. 1.7. 1.5 

Type 

3 

Identi fy 

r 13 

2. 2. 1.7 

Type 

2 

Classify 

Target  Classification 

- 41 

2. 2. 1.7.1 

Type 

2 

Perform  al 1 classi- 
fication functions 
to  identify 

2. 2. 1.7. 1.6 

Type 

2 

Determine  as  to 

42 

2. 2. 1.7. 1.6. 2 

Type 

2 

Determine  if 

r 13 

2. 2. 1.7 

Type 

2 

Classi fy 

Target  Classification 

- 41 

2. 2. 1.7. 2 

Type 

2 

Perform  all  classi- 
fication functions 
to  identify 

-.  43 

2. 2. 1.7. 2.1 

Type 

3 

Identi fy 
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Task  No. 

Type 

Keyword 

Task  Area 

13 

2. 2. 1.7 

Type  2 

Classify 

Target  Classification 

41 

2.2.1 .7.2 

Type  2 

Perform  all  classi- 
fication functions 
to  identify 

43 

2.2.1 .7.2.3 

Type  2 

Report  probability 

43 

2. 2. 1.7. 2. 3.1 

Type  2 

Determine  the  Prob. 

13 

2. 2. 1.7 

Type  2 

Classify 

Target  Classification 

41 

2. 2. 1.7. 3 

Type  2 

Perform  all  ..lassi- 
fication  functions 
to  identify 

44 

2. 2. 1.7. 3.1 

Type  2 

Classify 

44 

2. 2. 1.7. 3. 1.3 

Type  2 

Classify 

15 

2.2. 1.1. 1.1 

Type  2 

Determine  if 

Flight  Gear  Inspection 

15 

2.2. 1.1. 1.2 

Type  2 

Determine  if 

Flight  Gear  Inspection 

15 

2.2. 1.1. 1.2.1 

Type  2 

Determine  if 

15 

2. 2. 1.1. 1.3 

Type  2 

Determine  if 

Flight  Gear  Inspection 

15 

2. 2. 1.1. 1.3.1 

Type  2 

Determine  if 

30 

2.2. 1 .2. 1 

Type  2 

Determine  if 

Mission  Safety 

20 

2. 2. 1.2. 1.1 

Type  3 

Determine 

20 

2. 2. 1.2.1 

Type  2 

Determine  if 

Mission  Safety 

20 

2. 2. 1.2. 1.2 

Type  3 

Determine 

22 

2.2. 1 .2. 3.:. 2 

Type  2 

Determine 

Interior  Inspection 

22 

2.2.1 .2. 3. 2. 3 

Type  2 

Determine 

SAR  Inspection 

22 

2. 2. 1.2. 3. 2. 3.1 

Type  2 

State  if 

24 

2. 2. 1.3. 1.1 

Type  3 

Identify  each 

^restart  Inspection 

24 

2. 2. 1.3.  i.2 

Type  2 

Determine 

Prestart  Inspection 

30 

2.2.1 . > . 2 . 2 

Type  3 

I dent  i f y 

Radar  Operation 

30 

2.2. 1 .3.2.3 

Type  3 

Determine 

Duration  Estimation 

30 

2. 2. 1.5. 2. 4 

Type  3 

Identify 

Elevation  & Obstruction 

30 

2. 2. 1.5.  .6 

'.Type  3 

Determine 

Magnetic  Bearing 

33 

2. 2. 1.6. 2. 2 

Type  3 

State  the  effects 

MAD  Operation 

33 

2.7.1  6.2  4 

Type  3 

Determine 

R0-32  Recorder 

Mai  function 

4 

3 


i 
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Task  No. 

Type 

Keyword 

Task  Area 

33 

2. 2. 1.6. 2. 5 

Type  3 

Determine 

ASQ-81  System 

Mai  function 

33 

2.2. 1 .6.2.6 

Type  3 

Identify 

Tactics,  Initial  Contact 

34 

2. 2. 1.6. 2. 1.1 

Type  3 

State 

ASQ-81  System  Operation 

34 

2. 2. 1.6. 2. 1.3.1 

Type  3 

Identi fy 

ASQ-81  System  Operation 

34 

2. 2. 1.6. 2. 1.4 

Type  3 

Identi fy 

ASQ-81  System  Operation 

36 

2. 2. 1.6. 3. 3.1 

Type  3 

State 

LN-66  Radar  Operation 

36 

2. 2. 1.6. 3. 3. 2 

Type  3 

State 

LN-66  Radar  Operation 

36 

2.2.1 .6. 3. 3. 3 

Type  3 

State 

LN-66  Radar  Operation 

37 

2. 2. 1.6. 4 1.1.2 

Type  3 

State 

ASA-26  System  Operation 

39 

2. 2. 1.6. 5.1 

Type  2 

State  the  best 

Visual  Search 

46 

2. 2. 1.8. 1.1 

Type  3 

Identify 

Tactics,  Tracking  with 
MAD 

47  * 

2. 2. 1.8. 2. 1.1. 2 

Type  3, 
Type  3 

Determine,  select 

Tactics,  Master  Sono- 
buoy  Selection 

47 

2.2.1 .8.2.1 .1 .3 

Type  3, 
Type  3 

Determine,  select 

Tactics,  Slave  Sonobuoy 
Selection 

47 

2.2.1 .8.2.1 . 1.4.1 

Type  2 

Determine 

Percentage  of 

Attenuation 

47 

2. 2. 1.8. 2. 1.1. 4. 3 

Type  2 

Determine 

Percentage  of 

Attenuation 

48 

2. 2. 1.8. 2. 2.1 

i-Type  2 

Determine 

Range  of  Target 

55 

2. 2. 2. 1.2.1 

i.Type  2 

List  all  info, 
pertaining 

SAR  Mission 

55 

2. 2. 2. 1.2. 2 

LType  2 

List  all  info, 
pertaining 

SAR  Mission 

55 

2. 2. 2.1 .2.3 

ilType  2 

List  all  info, 
pertaining 

SAR  Mission 

55 

2. 2. 2. 1.2. 4 

r.Type  2 

List  all  info, 
pertaining 

SAR  Mission 
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Task  No. 

Type 

Keyword 

Task  Area 

68 

2. 2. 2. 7. 4. 1.4.1 

Type  3 

Identi fy 

Survival  Hold  Release 

70 

2.2.2.12.1 

ZType  2 

List  all  info. 

SAR  Mission 

pertaining 

70 

2.2.2.12.2 

..Type  2 

List  all  info. 

SAR  Mission 

pertain ing 

70 

2.2.2.12.3 

ZType  2 

List  all  info 

SAR  Mission 

pertaining 

72 

2. 2. 3. 1.1 

ZType  2 

List  the  info. 

SAR  Mission 

75 

2. 2. 4. 8 

Type  3 

Identify 

Vessel  Classification 

’ 

80 

2. 2. 4.8. 1 

Type  3 

Identi fy 

* 

80 

2. 2. 4. 8. 1.1 

ZType  2 

List  the  critical . . 

r 

75 

2. 2. 4. 8 

Type  3 

Identify 

Vessel  Classification 

* 

80 

2. 2. 4. 8. 2 

Type  3 

Identify 

b 

80 

2. 2. 4. 8. 2.1 

Type  3 

Identify 

u 

80 

2. 2. 4. 8. 2.1  .1 

ZType  2 

l. i st  the  cri  tical . . 

r~ 

75 

2. 2. 4. 8 

Type  3 

Identify 

Vessel  Classification 

b 

80 

2. 2. 4.8. 2 

Type  3 

Identi fy 

b 

80 

2. 2. 4. 8. 2. 2 

Type  3 

Identify 

u 

80 

2. 2. 4. 8. 2. 2.1 

ZType  2 

State  the  critical . 

75 

2. 2. 4. 8 

Type  3 

Identi fy 

Vessel  Classification 

■' 

80 

2. 2. 4. 8. 3 

Type  3 

Identify 

* 

80 

2. 2. 4. 8. 3.1 

Type  3 

J den  t i fy 

80 

2. 2. 4. 8. 3. 1.1 

ZType  2 

List  the  critical . . 

75 

2. 2. 4. 8 

Type  3 

Identi fy 

Vessel  Classification 

• * 

80 

2. 2. 4. 8. 3 

Type  3 

Identify 

80 

2. 2. 4. 8. 3. 2 

Type  3 

I den  t i fy 

80 

2. 2. 4. 8. 3. 2.1 

ZType  2 

List  the  critical . . 

76 

2. 2. 4. 6. 1 

Type  3 

Identi fy 

Visual  Rig 

i* 

77 

2. 2. 4. 6.1 .1 

Type  3 

Identi fy 

■ » 

77 

2. 2. 4. 6. 1.1.1 

Type  2 

State  rules  for 

deter. 

r 

76 

2. 2. 4. 6.1 

Type  3 

Identify 

Visual  Rig 

* 

77 

2. 2. 4. 6. 1.2 

Type  3 

Identi fy 

77 

2. 2. 4. 6. 1.2.1 

Type  2 

Identify 
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Task  No. 

Type 

Keyword 

Task  Area 


r- 

76 

2. 2. 4. 6.1 

Type  3 

*■ 

77 

2.2.4.6.1 .2 

Type  3 

+■ 

77 

2. 2. 4. 6. 1.2.2 

Type  3 

~ 

76 

2. 2. 4. 6.1 

Type  3 

* 

77 

2.2.4.6.1 .2 

Type  3 

77 

2. 2. 4. 6.1 .2.3 

Type  2 

76 

2. 2. 4. 6.1 

Type  3 

' 

77 

2.2.4.6.1 .2 

Type  3 

u 

77 

2. 2. 4. 6. 1.2. 4 

Type  3 

r 

76 

2. 2. 4. 6.1 

Type  3 

f 

77 

2. 2. 4. 6. 1.3 

Type  3 

77 

2. 2. 4. 6.1 .3.1 

Type  3 

76 

2. 2. 4. 6.1 

Type  3 

77 

2. 2. 4. 6. 1 .4 

Type  3 

f 

77 

2. 2. 4. 6. 1.4. 2 

Type  2 

77 

2. 2. 4. 6. 1.4. 2.1 

iType  2 

r 

76 

2. 2. 4. 6.1 

Type  3 

.. 

77 

2. 2. 4. 6. 1.4 

Type  3 

:* 

77 

2. 2. 4. 6.1 .4.3 

Type  2 

* * 

77 

2. 2. 4. 6. 1.4. 3 

Type  2 

r 

76 

2. 2. 4. 6.1 

Type  3 

77 

2. 2. 4. 6. 1.4 

Type  3 

i , 

77 

2. 2. 4. 6. 1.4. 4 

Type  2 

r 

76 

2. 2. 4. 6. 1 

Type  3 

77 

2. 2. 4. 6. 1.4 

Type  3 

1 * 

77 

2. 2. 4. 6. 1.4. 5 

Type  3 

- 

76 

2. 2. 4. 6.1 

Type  3 

' 

77 

2. 2. 4. 6. 1.5 

Type  2 

r. 

76 

2. 2. 4. 6.1 

Type  3 

78 

2. 2. 4. 6. 1.6 

Type  2 

82 

2. 2. 5. 1.2.1 

i-Type  2 

82 

2. 2. 5. 1.2. 2 

J.Type  2 

Identify 

Identify 

Identi fy 

Visual 

> 

Rig 

Identify 

Identify 

Identify 

Visual 

Rig 

Identify 

Identify 

Identi fy 

Visual 

Rig 

Identify 

Identify 

Identify 

Visual 

Rig 

Identi fy 

Identi fy 

Identi fy 

State  the  critical . 

Identify 

Identify 

Identify 

State  rules  for 
identifying 

Visual 

Rig 

Identi fy 

Identi fy 

Identify 

Visual 

Rig 

Identify 

Identify 

Identify 

V isual 

Rig 

Identi fy 

Identify 

Visual 

Rig 

Identify 

Identify 

Visual 

Rig 

List  the  info, 
pertaining. . . 

Cargo  Mission 

List  the  info, 
pertaining. . . 

Cargo  Mission 
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Page 

Task  No. 

Decision 

Type 

82 

2. 2. 5. 1.2. 3 

FType  2/ 

82 

2.2.5. 

.1.2.4 

1-Type  4 

89 

2. 2. 7. 1.2.1 

>'■  Type  2 

89 

2. 2. 7. 1.2. 2 

j.Type  2 

89 

2. 2. 7. 1.2. 3 

?:Type  2 

89 

2.2.7. 

1.2.4 

;:Type  2 

90 

2. 2. 7. 6. 2 

Type  2 

93 

2. 3. 1.1. 3.1 

Type  2 

93 

2.3.1. 

1.3.2 

JiType  3 

93 

2.3.1. 

1.4 

Type  2 

93 

2.3.1 . 

2.2 

Type  2 

93 

2.3.1 .3.2 

Type  2 

Keyword 

Task  Area 

List  the  Info, 
pertaining. . . 

Cargo  Mission 

List  the  Info, 
pertaining. . . 

Cargo  Mission 

List  all  ... 
pert.aininq. . . 

Passenger  Transfer 
Mi ssion 

List. . . needed  by. . . 

Passenger  Transfer 
Mission 

List  info. . . that 
must . . . 

Passenger  Transfer 
Mission 

List  info. . . that 
must . . . 

Passenger  Transfer 
Mission 

Determine  if 

Flight  Equipment 
Inspection 

Determine  whether 

Ordnance  System 
Inspection 

State 

Ordnance  System 
Inspection 

Determi ne 

Ordnance  System 
Inspection 

Determine 

Ordnance  System 
Inspection 

Determine  if 

Ordnance  System 
Inspection 
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APPENDIX  C 

\ PILOT/ATO  (TS) 

Analysis  of  PILOT/ATO  Tape  Slides 


Slide  Old  NA/ND  Hidden/ 


Task  No. 

Task  Name 

No. 

Slide  Na 

DM/DE 

Obvious 

Type 

Keyword 

Fuel  System  Mai  - 
function  Detection 

F-l.5.1 

SF-1. 7.1 

NA 

1.1.10.1.4.1.1.1 

Boost  Pump 

Mai  function 

F -1 .5.2 

SF-1.7.3 

DE 

identi fy 

1.1.10.1.4.2.1.1 

Transfer  Pump 

Mai  function 

F -1 .5.3 

SF-1.7.4 

NO 

state 

1.1.10.1.4.3.1.1 

Compressor 

Malfunction 

F -1 .5.4 

SF-1.7.5 

ND 

1.1.4.3.12.1.1 

Bypass  Filter 
Malfunction 

F -1 .5.5 

SF - 1 .7.6 

NO 

state 

1.1.10.1.4.4.1.1 

Emergency  Pump 
Malfunction 

F-l  .5.6 

SF - 1 .7.7 

DM 

obvious 

Type  2 

whether 

or 

1 .1 .12.1 .3. 4.5.1 

External  Engine  Fire 

F- 1 .5.7* 

SF- 1 .7.8 

DE 

identi fy 

1.1.12.1.3.4.4.1 

Internal  Engine  Fire 

F-l. 5. 8 

SF-1 .7.9 

DE 

identify 

Internal  Aircraft 

Fire 

F-l. 5. 10* 

SF - 1 .7.11 

ND 

identify 

Oi 1 System  Malfunc- 
tion Detection 

F-l. 4.1 

SF-1. 5. 2 

NA 

1.1.10.1.5.2.1.1 

Gear  Box  Oil 
Malfunction 

F-l. 4. 2* 

SF-1. 5. 3 

DM 

obvious 

Type  2 

identify 

1.1.10.1.5.3.1.1 

Engine  Oil 

Malfunction 

F-l. 4. 3 

SF-1. 5. 4 

DE 

identify 

1.1.10.1.2.3.1.1 

Intermediate  Gear 

Box  Oil  Malfunction 

F-l. 4. 4 

SF-1. 5. 5 

ND 

state 

1.1.10.1.5.2.1.1 

Combining  Gear  Box 

Oil  Malfunction 

F-l. 4. 5* 

SF-1. 5. 7 

DM 

hidden 

Type  2 

identify 

1.1.10.1.5.1.2 

Speed  Decreaser 

F-l. 4. 6 

SF-1. 5. 7 

DE 

identify 

Hydraulic  System  Mal- 
function Detection 

F-3.5.1 

SF-2.5.1 

NA 

1.1.10.1.7.1 

Hydraulic  System 
Malfunction 

F-3.5.2* 

SF-2.5.3 

DM 

hidden 

identify 

ASE  Malfunction 

F-3.3.1 

SF-2.3.1 

NA 

Detection 

★ 

See  notes  which  follow  this  appendix 
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Task  No. 

Task  Name 

Slide 

No. 

Old 

Slide  Na 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type 

Keyword 

1.1. 4. 3. 1.1.1 

ASE  Malfunction 

F-3.3.2 

SF-2.3.3 

DE 

identify 

Landing  Gear  Failure 
to  Lower 

F-3.3.3 

SF-2.3.5 

DE 

identify 

1.1.4.3.14.1.1 

Tire  Blowout 

F-3.3.4 

SF-2.3.6 

ND 

state 

1.1.12.1.3.6.2 

Postflight  Inspection 

F-4.2.4 

PE-3A.1.4 

ND 

name 

1.1.12.1.3.4.3.1 

Droop  Stops  Failure 

F-4 . 1 . 1 

PF-3.1.1 

DE 

identify 

Engine  Malfunction 
Detection 

F-5.3.1 

SF-4.3.2 

NA 

1.1.10.1.1.1.1 

Engine  Flameout 

F-5.3.2* 

SF-4.3.3 

DE 

identify 

1.1.10.1.1.2.1.1 

Throttle-Caused 

Power  Osci 1 lation 

F-5.3.3* 

SF-4.3.4 

DE 

identify 

1.1.10.1.1.4.1.1 

Partial  Power  Loss 

F-5.3.4 

SF-4.3.6 

DE 

identify 

1.1.10.1.1.5.1.1 

Compressor  Stal 1 

F-5.3.5* 

SF -4.3.7 

DE 

identi  fy 

1.1.10.1.1.6.1.1 

Electrically-Caused 

Overspeed 

F-5.3.6* 

SF-4.3.8 

DM 

hidden 

Type  2 

identify 

1.1.10.1.1.7.1.1 

Loss  of  Nf  Signal 

F-5.3.7 

SF-4.3.9 

DE 

identify 

Flight  Control  Mal- 
function Detection 

F-5.2.1 

SF-4.2 

NA 

1.1. 4. 3. 9. 1.1 

Loss  of  Tail  Rotor 
Thrust 

F-5.2.2* 

SF-4.2.3 

DM 

hidden 

Type  2 

identify 

1.1.4.3.15.1.1 

Loss  of  Tail  Rotor 
Control 

F-5.2.3* 

SF-4.2. 4 

DM 

hidden 

Type  2 

identify 

Elec' 1 System  Mal- 
function Detection 

F-7.3.1 

SF-5.3.1 

NA 

1.1.4.3.11.1.1.1 

Generator 

Ma 1 function 

F-7.3.2* 

SF-5.3.3 

DE 

identi fy 

1.1.4.3.11.4.1.1 

Converter 

Malfunction 

F-7.3.3 

SF-5.3.5 

DE 

identify 

1.1.4.3.11.3.1.1 

Total  Electrical 

Fai lure 

F-7.3.4* 

SF-5.3.6 

DE 

identify 

26-Volt  AC  Trans- 
former Failure 

F-7.3.5 

SF-5.3.7 

DE 

identi fy 

See  notes  which  follow  this  appendix 
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Task  No. 

Task  Name 

Slide 

No. 

Old 

SI  ide  Na 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type 

Keyword 

1.1. 4. 3. 3. 1.1 

Torque  Gauqe 

Ma 1 func t ion 

F-7.3.6 

SF-S.3.10 

DE 

ident i fy 

1.1.12.1.3.4.2.1 

Shutdown  with  Total 
Electrical  Failure 

F-7.3.7 

SF-5.3.11 

NO 

state 

1.1. 4. 3. 8. 1.1 

Dual  Enqine  Fai lure 

F-7.2.1 

SF-5.2.1 

ND 

state 

1.1. 4. 3. 7. 1.1 

Sinqle  Enqine 

F a i 1 u re 

F-7.2.2* 

SF -5.2.2 

DE 

identify 

Electrical  Fire 
Detection 

F-7.2.4 

SF-5.2.6 

DE 

identify 

1.1.10.2.3.1 

Determining  Restart 
Feasibi 1 ity 

F-7.1.2* 

SF-5.1.2 

DM 

obvious 

Type  2 

decide 

whether 

1.1. 4. 3. 4. 1.1 

Decaying  Nr/Nf 

F-9.2.1 

SE-7.2.1 

DE 

identify 

1.4. 6. 1.3. 3. 1.1 

Settl inq  wi th  Power 

F-9.2.2 

SU-1.1.1 

ND 

define, 

name, 

state 

1.4. 6. 1.3. 1.1 

Power  Settling 

F-9.2.3 

SU- 1.1.3 

DE 

identify 

1.4. 6. 1.3. 2. 1.1 

Blade  Stal 1 

F- 1 1 .1.1* 

SF-8.1.1 

DE 

identify 

1.1.12.1.3.4.1.2 

Locked  Tailwheel 

F-ll.1.2 

ND 

state 

★ 

See  notes  which  follow  this  appendix 
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PILOT/ATO  (TS) 


Slide  No.  Notes 

F- 1 .5.7  Measure  of  criticality  of  a mission  is  considered  by  the  pilot. 

This  information  is  used  to  decide  whether  to  continue  the 
mission  or  to  abort. 

F-l.5.10  Identify  cues  and  alerts  for  cabin  fire  by  fumes,  visible  smoke, 

and/or  fire. 

F- 1 .4.2  The  cues  and  symptoms  are  given  and  the  pilot  must  decide 

whether  or  not  there  is  a gearbox  malfunction.  Phrases  such  as 
"the  first  indication  of  malfunction  in  the  oil  system  will 
probably  be  the  transmission  oil  pressure  caution  light"  are 
useiT  As  it  is  indicated  in  the  tape  slide,  this  is  a "no  ask 
from  ground"  procedure. 

F- 1 .4.5  Possible  symptoms  for  a combining  gearbox  oil  malfunction  are 

discussed.  The  pilot  must  identify  the  malfunction.  Phrases 
such  as  "a  complete  break  in  the  oil  line  will  probably  cause 
these  indications  (1)  the  combining  gear  box  caution  light  will 
illuminate,  (2)..."  are  used. 

F-3.5.2  Phrases  such  as  "the  hydraulic  pressure  caution  light  in  the 

caution  light  panel  will  probably  be  your  first  indication  of  a 
malfunctioning,  but  remember  a caution  light  could  be  accom- 
panied by  a sudden  burst  of  flight  in  the  event  of  a massive 
break  in  the  line"  are  used. 

F-5.3.2  The  reaction  speed  is  critical.  The  previously  made  decision  is 

communicated  to  the  pilot  for  execution  via  manuals  and  gauges. 

F-5.3.3  Determine  whether  it  is  throttle-caused  oscillation  or  engine- 

caused  oscillation.  The  procedure  to  test  and  determine  is 
given. 

F-5.3.5  Symptoms  are  given  but  the  procedure  to  obtain  a diagnosis 

through  the  use  of  those  symptoms  is  not  mentioned. 

F-5.3.6  Phrases  such  as  "on  the  other  hand,  if  the  aircraft  is  on  the 

descent  or  on  deck,  faulty  electrical  input  to  the  fuel  control 
will  probably  result  in  rotor  overspeed.  The  symptoms  for 
electrically  caused  rotor  overspeed  are:  (1)  NFNR  for  .... 
(2)..."  are  used. 
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Slide  No.  Notes 


F-5.2.2 


F-5.2.3 


F-7.3.2 


1 

F-7.3.4 


F-7.2.2 


F-7.1 .2 


F- 7 . 11.1.1 


As  symptoms,  phrases  such  as  "...  depending  upon  flight 
condition,  aerodynamic  coupling  may  produce  right  roll  and 
pitch  down.  The  rate  of  yaw  will  depend  upon  the  amount  of 
torque  applied  to  the  ..."  are  used.  It  may  be  critical 
because  it  is  said  that  "loss  of  tail  rotor  thrust  demands 
immediate  action".  It  may  cause  total  loss  of  control  which, 
in  turn,  may  cause  landing  on  the  nose.  Also  there  are  phrases 
such  as  "(1)  begin  a 70  Ng  minimum  power  descent  or  autorotation 
depending  upon  the  intensity  of  the  vibration". 

Some  alternatives  are  given  for  yaw  controlling  the  aircraft 
with  loss  of  tail  rotor  control,  but  the  criteria  for  choosing 
any  one  of  them  is  not  mentioned. 

The  pilot  must  figure  out  how  much  gas  is  left  without  the  use 
of  working  instruments  (due  to  lack  of  generator). 

Same  as  F-7.3.2 


Phrases  are  used  such  as  "the  first  indication  of  engine  failure 
will  probably  be  the  sound  of  the  engine  unwinding.  This  will 
be  followed  by  . . . " . 


Whether  or  not  to  attempt  a restart  is  investigated.  Phrases 
are  used  such  as  "the  ultimate  decision  as  to  whether  or  not  a 
restart  should  be  attempted  rests  with  the  pilot  in  command. 

In  making  this  decision  he  must  rely  on  his  knowledge  of  NATOPS, 
his  good  judgment  and  his  personal  experience.  There  are  no 
hard  and  fast  rules  of  attempting  to  restart  after  an  engine  is 
flamed  or  has  been  secured  in  flight.  But  there  are  some 
general  guidelines  which  can  be  followed.  An  engine  restart 
should  be  attempted  under  the  following  conditions:  (1)..., 
(2)...,  (3)...  Remember,  do  not  attempt  to  restart  if  the  engine 
is  in  flame  because  of  an  internal  or  external  fire." 


Critical . 


Requires  rapid  identification  and  correction  action. 
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APPENDIX  D 
CREW  (TS) 

Analysis  of  Crewman  Tape  Slides 


Task  Name 

Old 

Slide  No . 

NA/ND  - 

Keyword 

Planned  Ditching 

CRF-1 .3.4 

ND 

state 

Search  & Rescue  Equipment 

CRF-1 .5. 5/6 

ND 

name,  state 

Crewman  Safety  Belt 

CRF-1. 6. 3 

ND 

describe,  name 

Ai rcraft  Handl ing  #1 

PF-3.2.1 

ND 

state 

Aircraft  Handling  #2 

P-:-3.2.2 

ND 

state 

Aircraft  Movement  Safety  Rules 

PF-3.2.4 

ND 

state 

Signature  Cards 

PF-2.1.3 

ND 

demonstrate 

Immediate  Ditching  Procedure 

CRF-1 .3.3 

ND 

state 

Bailout  Procedure 

CRF-1 .3.5 

ND 

st^te 

Total  Electrical  Failure 

CRF-1 .3.7 

ND 

identify 

Internal  & External  Engine  Fire 

CRF-1 .3.9 

ND 

state 

Hand  Pressure  Signals 

CRH-1.3.3 

ND 

state 

Cabin  and/or  Electrical  Fire 

CRF-1. 3. 11 

ND 

state 

Hung  Droop  Stop 

CRF-1. 3. 14 

ND 

state 

Cargo  Transfer  Safety  Precaution 

CRH-1.4.1 

ND 

state 

Cargo  Rigging  Safety  Precaution 

CRH-1.4.5 

ND 

state 

Normal  Hoisting  Operations 

CRH-1.3.7 

ND 

state 

Connect/Disconnect  Refueling  Hose 

CRH-2.1.4 

ND 

state 

Passenger  Survival  Gear 

CRH-2.1.4 

ND 

name 

Swimmer's  Equipment 

CRW-1.2.1 

ND 

identify,  state 

Required  Flight  Gear 

CRF-1. 6. 7 

ND 

name 

Regaining  Lost  UHF  Communication 

SE-4.9.2 

NO 

name 

LAMPS  Frequency  Circuits 

SE-4.9.3 

ND 

name 

Aircraft  Positioning  During  HIFR 

CRH-2.1.2 

ND 

state 
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Task  Name 

Old 

Slide  No. 

NA/ND 

Keyword 

Sonobuoy  Loading  Safety 

OR-2.1.8 

ND 

state 

Inspecting  JAU-1  B CAD 

OR- 1 .1.5 

ND 

state 

Inspecting  Sonobuoy  Launch 
Container 

OR- 1 .1.3 

ND 

name 

Major  Preflight  Inspection  Areas 

PF-3A.2.7 

ND 

name 

Radar  Interpretation  - 
Land  Mass 

NA-2.1 .1* 

ND 

name 

Radar  Interpretation  - 
Heavy  Weather  Contact 

NA-2.1. 2* 

ND 

name 

Radar  Interpretation  - 
Surface  Contact 

NA-2.1 .3* 

ND 

name 

Radar  Interpretation  - 
Airborne  Contacts 

NA-2.1. 4* 

ND 

name 

Sound  Velocity 

AC-1.2.11 

ND 

describe 

*See  notes  which  follow  this  appendix. 
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CREW  (TS) 


Slide  No.  Notes 


NA- 2 .1.1  This  slide  prepares  the  crewmen  for  distinguishing  between  a 

target  and  a land  mass  which  itself  is  a classification  and, 
therefore,  a decision  task.  Only  the  attributes  (features  of  a 
land  mass  on  radar  (one  of  the  two  classes)  is  listed  and  the 
crewman  is  supposed  to  classify  the  object  as  a land  mass  if 
he  identifies  those  attributes. 

NA-2.1.2  "Name  three  radar  display  characteristics  of  heavy  weather"  is 

the  objective  of  this  course.  These  characteristics  (features) 
are  listed. 

NA-2.1.3  The  three  radar  display  characteristics  of  surface  contact  are 

1 i sted. 

NA-2.1.4  The  three  radar  display  characteristics  of  an  airborne  contact 

are  listed.  One  of  them  (fast  movement)  is  mentioned  to  be  the 
distinguishing  characteristics  between  airborne  contact  and 
surface  contact. 
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APPENDIX  E 
PILOT/ AJOjt  WB ) 

Analysis  of  P I LOT / ATO  Workbooks 


Task  No. 

Task  Name 

Section 

No. 

1.1.10.1.2.4.1.3 

Transmission  System 

F - 1 .8.1 

1.1.10.1.2.4.1.2 

II 

F- 1 .8.2 

1.1.10.1.2.4.3 

II 

F-1.R.3 

1.1.10.1.2.4.2 

II 

F-l .8.4 

1.1.10.1.2.4.1 

II 

F-l .8.5 

1.1.10.1.2.4.1.1 

II  « 

F-l .8.6 

Fire  Detection  and 
Extinguishing  System 

F-l. 7.1 

1.1.4.3.10.6.1.2 

11 

F-l 

.7.2 

1.1.4.3.10.6.3 

II 

F-l 

.7.3 

1.1.4.3.10.6.4 

II 

F-l 

.7.4 

1.1.4.3.10.6.2 

II 

F-l 

.7.5 

1.1.4.3.10.6.1 

" 

F-l. 7. 6 

1.1.4.3.10.6.1.1 

II 

F-l 

.7.7 

1.1.4.3.12.4.1.3 

Fuel  System 

Operation 

F-l. 6.1 

1.1.4.3.12.4.1.2 

II 

F-l. 6. 2 

1.1.4.3.12.4.3 

II 

F-l. 6. 3 

1.1.4.3.12.4.4 

II 

F-l .6.4 

II 

F-l. 6. 5 

1.1.4.3.12.4.1 

II 

F-l. 6. 6 

1.1.4.3.12.4.1.1 

II 

F-l. 6. 7 

Fuel  Control 

Divider  Setting 

F-l. 5. 9 

1.1.10.1.6.6 

Flight  Control 

System 

F-l. 3.1 

1.1.12.1.3.2 

Disengage  Rotor 
Envelope 

F-l. 2. 7 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

SF-1 .10.1 

ND 

describe 

ND 

" 

ND 

•• 

ND 

II 

ND 

II 

ND 

" 

SF-1.9.'. 

ND 

describe 

ND 

II 

ND 

II 

ND 

II 

ND 

II 

ND 

II 

ND 

II 

SF-1. 8.1 

ND 

describe 

ND 

II 

ND 

II 

ND 

•1 

ND 

II 

ND 

II 

ND 

•I 

SF-1. 7.10 

ND 

state 

SF-1. 4.1 

ND 

describe 

SF-1. 3. 7 

DE 

determine 

if 

r 
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Task  No. 

Task  Name 

Sec  t i on 

No. 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

Airspeed 

F-l .2.6 

SF-1 .3.6 

DE 

calculate 

1.1.10.2.1.3 

Density 

F-l .2.5 

SF-1.3.5 

DE 

determine 

1.1.10.2.1.3 

Torque 

F-l .2.4 

SF-1. 3. 4 

DE 

determine 

Endurance 

F-l  .2.3* 

SF-1. 3. 3 

DE 

determine 

1.1. 2. 1.1 

Gross  Weight 

F-l. 2. 2 

SF-1. 3. 2 

DE 

calculate 

1.1. 2.1 

Single  Engine 

Airspeed 

F-l. 2.1 

SF-1. 3.1 

DE 

determine 

1.1.11.2.1.1 

VMC  Recovery  to  Land 

F-l. 1.4* 

SF-1. 1.4 

ND 

state 

Land  Recovery 

F-l. 1.3 

SF-1. 1.3 

ND 

state 

1.1. 4. 2. 1.1 

Normal  Take-Off 

F-l . 1 .2 

SF-1. 1.2 

ND 

state 

1.1. 4. 2. 1.7.1 

Taxi  Procedure 

F-l. 1.1 

SF-1. 1.1 

ND 

state 

Airframe  System 

F-3.  11.1 

SF-2.11.1 

ND 

describe 

1.1.10.6.3.1.2 

F-3.11.2 

ND 

» 

1.1.10.1.6.3.3 

" 

F-3. 11. 3 

ND 

» 

1.1.10.1.6.3.2 

II 

F-3. 11. 4 

ND 

II 

1.1.10.1.6.3.1 

'• 

F-3. 11. 5 

ND 

II 

1.1.10.1.6.3.1.1 

II 

F-3. 11. 6 

ND 

II 

1.1.10.1.6.3.1 

Rotor  System 
Operation 

F-3. 10.1 

SF-2.10.1 

ND 

describe 

1.1.10.1.6.3.1.2 

II 

F-3. 10.2 

ND 

II 

1.1.10.1.6.3.3 

» 

F-3. 10. 3 

ND 

II 

1.1.10.1.6.3.2 

Rotor  System 
Operation 

F-3. 10.4 

ND 

describe 

1.1.10.1.6.3.1 

» 

F-3. 10. 5 

II 

II 

1.1.10.1.6.3.1.1 

II 

F-3. 10.6 

II 

» 

1.1.10.1.6.4.1.2 

Wheel  Brakes  System 

F-3. 9.1 

SF-3.9.1 

ND 

describe 

.. 

F-3. 9. 2 

II 

II 

★ 


See  notes  which  follow  this  appendix 
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Task  No. 

Task  Name 

Section 

No. 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

1.1.10.1.6.4.  3 

Wheel  Brakes  System 

F-3.9.3 

ND 

describe 

1.1.10.1.6.4.3 

M 

F-3.9.4 

II 

II 

1.1.10.1.6.4.2 

II 

F-3.9.5 

II 

•• 

1.1.10.1.6.4.1 

•• 

F-3.9.6 

II 

II 

1.1.10.1.6.4.1.1 

“ 

F-3.9.7 

II 

“ 

Blade  Track  System 

F-3.8.1 

SF-2.8.1 

ND 

describe 

•I 

F-3.8.2 

II 

II 

1.1.10.1.6.9.4 

II 

F-3.8.3 

II 

•I 

F-3.8.4 

II 

•• 

1.1.10.1.6.9.1 

Blade  Track  System 

F-3.8.5 

ND 

describe 

1.1.10.1.6.9.1.1 

II 

F-3.8.6 

II 

11 

1.1.10.1.7.2.1.3 

Hydraulic  System 

F-3.6.1 

SF-2.6.1 

ND 

describe 

1.1.10.1.7.2.1..' 

ii 

F-3.6.2 

II 

II 

1.1.10.1.7.2.3 

•• 

F-3.6.3 

II 

II 

1.1.10.1.7.2.4 

li 

F.3.6.4 

II 

11 

1.1.10.1.7.2.2 

ii 

F-3.6.5 

II 

II 

1.1.10.1.7.2.1 

il 

F-3.6.6 

II 

II 

1.1.10.1.7.2.1.1 

II 

F-3.6.7 

II 

II 

1 . 1 . 4 . 3.  l.l.l.l.l . 2 

ASE  System 

F-3.4.1 

SF-2.4.1 

ND 

describe 

1 . 1 . 4. 3.  l.l.l.l . 3 

II 

F-3.4.2 

II 

II 

1.1. 4. 3. 1.1. 1.4 

•• 

F-3.4.3 

II 

II 

1.1. 4. 3.  l.l.l.l. 2 

« 

F-3.4.4 

II 

•1 

1.1. 4. 3.  l.l.l.l.l 

II 

F- 3.4.5 

II 

II 

1 . 1 .4.3. 1.11.1.1.1 

II 

F -3 .4.6 

II 

" 

1.1. 3. 2. 4 

Battery  Start 

F-3.2.1 

SF -2.2. 1 

ND 

state 
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Task  No. 

Task  Name 

Section 

No. 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

Precision  Approach 

F-3.1. 1 

SF-2. 1.3 

NO 

s t.a  te 

Normal  Approach 

F-3.1.2 

SF-2.1.4 

NO 

state 

Rustl ick 

F-4.2.3 

PF-3A.1.  3 

ND 

state 

1.1.12.1.1.2 

Vids  Fill-Out 

F-4.2.2 

PF-3A.1. 2 

ND 

Fill  out 

1.1.12.1.1.1 

Yel low  Sheet 

F-4.2.1 

PF-3A.1. 1 

ND 

list 

1.8. 2. 3. 8. 1.1 

Navigation  System 

F-5.8.7 

ND 

describe 

1.8. 2. 3. 8.1 

II 

F-5.8.6 

II 

II 

1.8. 2. 3. 8. 2 

II 

F-S.8.5 

II 

II 

1.8. 2. 3. 8. 4 

II 

F-5.8.4 

II 

II 

1.8. 2. 3. 8. 3 

• l 

F-5.8.3 

II 

II 

1.8. 2. 3. 8. 1.2 

II 

F-5.8.2 

II 

II 

1.8. 2. 3. 8. 1.3 

II 

F-5.8.1 

SF-4.6.1 

II 

II 

1.8. 2. 3. 1.1.1 

Pitot  Airspeed 

System 

F-5.7.7 

ND 

describe 

1.8. 2. 3. 1.1 

•• 

F .5.7.6 

• 1 

II 

1.8. 2. 3. 1.2 

II 

F.5.7.5 

II 

II 

1.8. 2. 3. 1.3 

II 

F.5.7.4 

» 

II 

1.8. 2. 3. 1.3 

li 

F.5.7.3 

II 

II 

1.8. 2. 3. 1.1. 2 

II 

F-5.7.2 

II 

II 

1.8. 2. 3. 1.1. 3 

II 

F-5.7.1 

SF-4.7.1 

•1 

II 

1.1.10.1.1.8.1.1 

Power  Plant  System 

F-5.4.7 

NA 

1.1.10.1.1.8.1 

II 

F-5.4.6 

" 

1.1.10.1.1.8.2 

II 

F-5.4.5 

II 

1.1.10.1.1.8.4 

II 

F-5.4.4 

-• 

1.1.10.1.1.8.3 

11 

F-5.4.3 

II 

1.1.10.1.1.8.1.2 

II 

F-5.4.2 

•• 

1.1.10.1.1.8.1.3 

II 

F-5.4.1 

SF-4.5.1 

II 
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Task  No. 

Task  Name 

Section 

No. 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

1.1.11.2.3.2.1 

Recovery  w/  loss  of 
Tail  Rotor  Control 

F-5.1 .5* 

SF-7.1.2 

DE 

state 

1.1.10.2.2.2.1 

Running  Autorotation 
Landing 

F-5.1 .3 

SF-4.1.3 

ND 

state 

1.1.10.2.2.1.1 

Fuel  Flare  Auto- 
rotation  Landing 

F-5.1. 2 

SF-4.1.2 

NO 

state 

1.1.10.2.1.4 

Emergency  RPM  Con- 
trol Approach 

F-5.1 . 1* 

SF-4.1.1 

DE 

state 

Communications 

System 

F-7.7.6 

ND 

describe 

1.8. 2. 5. 5.1 

ll 

F-7.7.5 

II 

ll 

1.8. 2. 5. 5. 2 

" 

F- 7 .7.4 

11 

ll 

1.8. 2. 5. 5. 3 

ll 

F-7.7.3 

•• 

ll 

1.8. 2. 5. 5. 1.2 

" 

F-7.7.2 

ll 

1.8. 2. 5. 5. 1.3 

ll 

F-7.7.1 

SF-5.6.1 

II 

II 

Caution  Lights 

System 

F-7.6.6 

ND 

describe 

1.8. 2. 5. 5.1 

ll 

F-7.6.5 

II 

II 

1.8. 2. 5.5.2 

ll 

F-7.6.4 

II 

ll 

1.8. 2. 5. 5. 3 

ll 

F-7.6.3 

II 

ll 

1.8. 2. 5. 5. 1.2 

ll 

F-7.6.2 

II 

ll 

ll 

F-7.6.1 

SF -4.16.1 

II 

ll 

1.8. 2. 2.1 

Lighting  System 

F-7.5.5 

ND 

describe 

1.8. 2. 2. 4 

II 

F-7.5.4 

II 

ll 

1.8. 2. 2. 5 

ll 

F-7.5.3 

II 

11 

1.8. 2. 2. 2 

ll 

F-7.5.2 

II 

ll 

1.8. 2. 2. 3 

ll 

F-7.5.1 

SF-5.5.1 

'■ 

II 

* 

See  notes  which  follow  this  appendix 
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Task  No. 

Task  Name 

Section 

No. 

Old  Sec- 
tion No. 

NA/N0 

DM/DE 

Hidden/ 

Obvious 

Type 

Keyword 

1.1.4.3.11.5.1.1 

Electrical  System 

F-7.4.7 

ND 

describe 

1.1.4.3.11.5.1 

II 

F-7.4.6 

II 

II 

1.1.4.3.11.5.2 

II 

F-7.4.5 

II 

it 

1.1.4.3.11.5.4 

II 

F-7.4.4 

II 

II 

1.1.4.3.11.5.3 

II 

F-7.4.3 

II 

•l 

1.1.4.3.11.5.1.2 

II 

F-7.4.2 

II 

» 

1.1.4.3.11.5.1.3 

II 

F-7.4.1 

SF-5.4.1 

II 

II 

1.1.10.2.4.1 

Ditching  Station 

F-7.2.8* 

SF-5.2.11 

DE 

state 

1.1.10.2.4 

Immediate  Ditching 

F-7.2.7* 

SF-5.2.10 

DM 

hidden 

Type  2 

state 

1.1.10.2.4.1 

Controlled  Ditching 

F-7.2.6 

SF-5.2.9 

ND 

state 

1.1.10.2.6 

Water  Takeoff 

F-7.2.5 

SF-5.2.7 

ND 

state 

1.1.4.3.10.5 

Internal  A/C  Fire 
Likelihood 

F-7.2.3 

SF-5.2.3 

ND 

state 

1.1.11.1.3.2.3.3 

Single  Engine 
Recovery 

F-7.1.6 

SF-5.1 .6 

ND 

state 

1.1.11.1.3.2.3.1 

Jettisoning 

F-7.1.3 

SF-5.1.3 

ND 

state 

1.1.10.2.3.2 

Secured  Engine 
Restart 

F-7.1.1 

SF-5.1.1 

ND 

state 

1.8. 2. 4. 2. 1.1 

Weapons  Systems 

F-9.8.7 

NA 

1.8. 2. 4. 2.1 

II 

F-9.8.6 

II 

1.8. 2. 4. 2. 2 

II 

F-9.8.5 

II 

1.8. 2. 4. 2. 4 

II 

F-9.8.4 

II 

1.8. 2. 4. 2. 3 

" 

F-9.8.3 

II 

1.8. 2. 4. 2. 1.2 

• 1 

F-y.8.2 

II 

1.8. 2. 4. 2. 1.3 

Weapons  System 

F-9.8.1 

NA 

1.8. 2. 4. 2. 1.1 

Electrical  Release 
System 

F-9.7.7 

ND 

describe 

See  notes  which  follow  this  appendix 
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Task  No. 

Task  Name 

Section 

No. 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

1.8. 2. 4. 2.1 

Electrical  Release 
System 

F-9.7.6 

ND 

describe 

1.8. 2. 4. 2.1 

II 

F-9. 7.5 

II 

II 

1.8. 2. 4. 2. 4 

II 

F-9.7.4 

II 

II 

1.8. 2. 4. 2. 3 

II 

F-9. 7. 3 

II 

II 

1.8. 2. 4. 2. 1.2 

11 

F-9. 7.2 

II 

II 

1.8. 2. 4. 2. 1.3 

•l 

F-9. 7.1 

SF-7.10. 1 

II 

•' 

Sonobuoy  System 

F-9. 6. 3 

ND 

describe 

II 

F-9. 6. 2 

II 

II 

II 

F-9. 6.1 

SF-5.9. 1 

II 

1.8. 2. 4. 8. 1.1 

ALR-54  System 

F-9. 5. 7 

ND 

describe 

1.8. 2. 4. 8.1 

II 

F-9. 5. 6 

II 

•• 

1.8. 2. 4. 8. 2 

•l 

F-9. 5. 5 

II 

II 

1.8. 2. 4. 8. 4 

II 

F-9. 5. 4 

II 

II 

1.8. 2. 4. 8. 3 

II 

F-9. 5. 3 

•1 

•• 

1.8. 2. 4. 8. 1.2 

II 

F-9. 5. 2 

II 

II 

1.8. 2. 4. 8. 1.3 

II 

F-9. 5.1 

SF-7.4.1 

II 

II 

1.8. 2. 4. 7. 1.1 

MAD  System 

F-9. 4. 7 

ND 

describe 

1.8. 2. 4. 7.1 

II 

F-9. 4. 6 

•1 

II 

1.8. 2. 4. 7. 2 

II 

F-9. 4. 5 

II 

II 

1 .8. 2. 4. 7. 4 

II 

F-9. 4. 4 

II 

II 

1.8. 2. 4. 7. 3 

II 

F-9. 4. 3 

II 

It 

1.8. 2. 4. 7. 1.2 

II 

F-9. 4. 2 

II 

II 

1.8. 2. 4. 7. 1.3 

•1 

F-9. 4.1 

SF-7.3.1 

II 

II 

1.1. 6. 1.3. 6 

MAD  Use 

F-9. 3. 5* 

SE-1.4.4 

NA 

★ 

See  notes  which  follow  this  appendix 
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Task  No. 

Task  Name 

Section 

No. 

Old  Sec- 
tion No. 

NA/ND 

DM/DE 

Hidden/ 

Obvious 

Type  Keyword 

MAO  Pattern 

Strengths 

F-9.3.4 

SE-1.2.8 

NA 

MAD  Localization 

F-9.3.3* 

SE-1.2.7 

NA 

MAO  Procedures 

F-9.3.1 

SE-1.2.5 

ND 

state 

1.1. 4. 2. 3.1 

Max.  Gross  Weight 
Takeoff 

F-9.1.1 

SF-7.1.1 

ND 

state 

LN66  HP  Radar  System 

F- 1 1 .5.3 

NO 

describe 

1.8. 2. 3. 7.1 

II 

F- 11 .5.2 

II 

li 

1.8. 2. 3. 7. 2 

II 

F-ll.5.1 

SF-4.12. 1 

II 

" 

1.8. 4. 2. 1.1 

Cargo  Hook  System 

F-ll .4.6 

ND 

describe 

1.8. 4. 2.1 

II 

F-ll .4.5 

II 

II 

1.8. 4. 3 

II 

F-ll. 4. 4 

II 

II 

1.8. 4. 2. 2 

II 

F-ll. 4. 3 

II 

II 

1.8. 4. 2. 1.2 

11 

F-ll .4.2 

II 

II 

1.8. 4. 2. 1.3 

II 

F-ll. 4.1 

SF-8.4.1 

II 

II 

1.8. 5. 3. 1.1 

Environmental  System 

F-ll. 3. 7 

ND 

describe 

1.8. 5. 3.1 

•l 

F-ll. 3. 6 

II 

II 

1.8. 5. 3. 2 

II 

F-ll. 3. 5 

II 

II 

1.8. 5. 3. 4 

II 

F-ll. 3. 4 

II 

" 

1.8. 5. 3. 3 

II 

F-ll. 3. 3 

II 

II 

1.8. 5. 3. 1.2 

II 

F-ll. 3. 2 

II 

II 

1.8. 5. 3. 1.3 

II 

F-ll. 3.1 

SF-8.3.1 

II 

II 

Radar  Offset 

F-ll. 2. 2* 

SE-4.8.3 

DM 

obvious 

Type  2 state 

Approach 


See  notes  which  follow  this  appendix 
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PILOT/ATO  (WB) 


Section  No. 


F-l.2.3 


F-l.1.4 


F-5.1.5 


F-5.1.1 


F-7.2.8 


F-7.2.7 


F-9.3.5 


F-9.3.3 


Notes 

There  is  a hidden  decision  making  task  inside  the  whole  task 
"determine  the  required  fuel  reserve";  however,  the  procedure 
for  its  calculation  is  given  "10%  of  fuel  on  board  or  20  minutes 
of  flight  time,  whichever  is  longer". 

There  are  two  hidden  decision  making  tasks  inside  the  whole 
task : 

(1)  "Determine  destination  weather",  and 

(2)  "Determine  traffic  pattern", 

neither  of  which  possesses  a procedure.  However,  both  are  said 
to  be  "asked  from  the  tower".  Therefore,  they  are  non-decision 
tasks  with  respect  to  the  pilot. 

The  procedure  is  given  in  the  form  of  "If  (condition) , then 
{action)11 , which  is  the  translation  of  a table  look  up.  This 
is  a means  of  corrmuni eating  a decision  previously  made  to  a 
decision  executer. 

There  is  a hidden  decision  "determine  type  of  landing  to  be 
made";  however,  the  procedure  to  perform  this  determination  is 
also  given. 

This  task  can  be  considered  as  a decision  execution  task.  Since 
the  decision  as  to  "should  an  immediate  ditching,  planned 
ditching,  or  none  be  performed"  has  previously  been  made  for 
all  possible  states  of  nature,  the  result  of  this  decision  is 
now  being  communicated  to  the  pilot  for  execution. 

Item  B-2  of  the  procedure  says,  "Either  secure  engine  and  rotors 
(release  harness  and  abandon  aircraft)  or  follow  single  engine 
water  takeoff  procedure".  However,  it  has  not  been  suggested 
how  to  decide  between  these  two  alternatives.  Therefore,  this 
remains  as  a decision  task  to  be  performed  by  the  pilot. 

This  task  is  potentially  a very  good  technical  and  also  critical 
(in  terms  of  the  objective  of  the  mission)  decision  task.  The 
course  materials  are  not  found  in  the  workbook  pack.  (It  might 
be  classified. ) 

It  is  a potential  decision  task;  however, the  corresponding  course 
material  is  not  found  in  the  workbook  pack.  (It  might  be 
classified. ) 


F - 1 1 .2.2  The  item  2 of  the  procedure:  "Determine  if  a direct  or  offset 

approach  will  be  used"  is  a decision  task.  In  the  description  of 
this  item,  it  is  said  that  "The  ATO  will  decide  if  a direct  or 
offset  approach  will  be  used".  No  completely  specified  criteria 
for  this  selection  has  been  given. 
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APPENDIX  F 


Single  Engine  Failure  (I) 
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30/, 

One  Engine 


Not  Sufficien 
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APPENDIX  G 

DECISION  TREE  FOR  LAND/WAVE-OFF 


1 6b 
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Safe 


8b 
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APPENDIX  H 

SCENARIOS  FOR  PROTOCOL  ANALYSIS 

Consider  yourself  engaged  in  each  of  the  following  scenarios. 
Describe  what  might  happen  in  terms  of  the  possible  alternative  courses 
of  action,  possible  states  of  nature,  and  the  outcomes  associated  with 
each  combination  of  a state  of  nature  and  a course  of  action. 

A.  Engine  Quits 

(1)  You  have  a single  engine  failure  while  flying  over  land. 
One  engine  is  sufficient  for  the  weight  of  the  aircraft 
and  there  is  a safe  landing  site  in  the  area. 

(2)  You  have  a single  engine  failure  while  flying  over  land. 
One  engine  is  sufficient  for  the  weight  of  the  aircraft 
and  there  is  an  unsafe  landing  site  in  the  area. 

(3)  You  have  a single  engine  failure  while  flying  over  land. 
Or.e  engine  is  not  sufficient  for  the  weight  of  the 
aircraft  and  there  is  a safe  landing  site  in  the  area. 

(4)  You  have  a single  engine  failure  while  flying  over  land. 

One  engine  is  not  sufficient  ur  the  weight  of  the 

aircraft  and  there  is  an  unsafe  landing  site  in  the 
area. 

(5)  You  have  a dual  engine  failure  while  flying  over  land. 

(6)  You  have  a fuel  starvation  while  flying  over  land  and 

you  don't  have  enough  altitude  to  bail  out. 

(7)  You  have  fuel  starvation  while  flying  over  land  and  you 
have  enough  altitude  to  bail  out.  However,  the  site 

is  dangerous. 


174 


NAVTRAEQU I PCEN  77-C-0005-1 


(8)  You  have  fuel  starvation  while  flying  over  land  and  you 
have  enough  altitude  to  bail  out.  The  site  is  safe. 

(9)  You  have  a single  engine  failure  while  flying  over  water. 

One  engine  is  suffucient  for  the  weight  of  the  aircraft. 

(10)  You  have  a single  engine  failure  while  flying  over  water. 

One  engine  is  not  sufficient  for  the  weight  of  the  aircraft. 

(11)  You  have  a dual  engine  failure  while  flying  over  water. 

(12)  You  have  fuel  starvation  while  flying  over  water. 

B . Land  or  Wave-Off 

(1)  You  have  attempted  landing.  The  aircraft  carries  a heavy 
cargo  and  the  environmental  condition  is  good.  The  landing 
site  is  prepared. 

(2)  You  have  attempted  landing.  The  aircraft  carries  a heavy 
cargo  and  the  environmental  condition  is  good,  but  the 
landing  site  is  unprepared. 

(3)  You  have  attempted  landing.  The  aircraft  carries  a heavy 
cargo  and  the  environmental  condition  is  bad,  but  the 
landing  site  is  prepared. 

(4)  You  have  attempted  landing.  The  aircraft  carries  a heavy 
cargo  and  the  environmental  condition  is  bad.  There  is 
an  unprepared  landing  site. 
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(5)  You  have  attempted  landing.  The  aircraft  carries  a light 
cargo  and  the  environmental  condition  is  good.  The  landing 
site  is  prepared. 


(6)  You  have  attempted  landing.  The  aircraft  carries  a light 
cargo  and  the  environmental  condition  is  good,  but  the 
landing  site  is  unprepared. 


(7)  You  have  attempted  landing.  The  aircraft  carries  a light 
cargo  and  the  environmental  condition  is  bad,  but  the 
landing  site  is  prepared. 


(8)  You  have  attempted  landing.  The  aircraft  carries  a light 
cargo  and  the  environmental  condition  is  bad.  There  is  an 
unprepared  landing  site. 


C.  Ditching 

(1)  There  is  an  alert  which  suggests  an  emergency  ditching, 
but  you  are  over  the  land. 

(2)  There  is  an  alert  which  suggests  an  emergency  ditching 
and  you  are  over  the  water. 

(3)  There  is  an  alert  which  suggests  a planned  ditching,  but 
you  are  over  the  land. 

(4)  There  is  an  alert  which  suggests  a planned  ditching  and 
you  are  over  the  water. 
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In  the  protocol  analysis  performed,  based  on  the  above  scenarios, 
the  answer  to  the  following  questions  were  pursued. 

Single  Engine  Failure  - Over  Land 

1.  Do  you  always  know  whether  one  engine  will  be  sufficient  to  continue 
on  the  mission  or  not? 

2.  How  do  you  know?  (or)  Why  don't  you  know? 

3.  If  one  engine  is  sufficient,  do  you  still  look  for  a safe  la  iding 
site? 

A.  Does  the  importance  of  the  mission  enter  into  your  decision  on  what 
to  do? 

5.  At  what  point  do  you  decide  whether  to  try  to  land  or  to  continue? 

6.  If  you  find  a safe  landing  site,  what  makes  you  decide  whether  to 
land  or  to  continue? 

7.  If  you  continue,  do  you  tend  to  make  changes  in  the  objectives  of 
the  mission?  That  is,  if  the  mission  has  more  than  one  purpose, 
do  you  try  to  accomplish  some  but  not  others? 

8.  If  you  continue,  do  you  evaluate  your  chances  of  being  successful? 
How? 

9.  If  you  can't  find  a safe  landing  site,  do  you  continue  looking  for 
one  or  do  you  try  something  else? 
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10.  If  one  engine  is  r.ot  sufficient  to  continue  and  you  find  a safe 
landing  site,  do  you  attempt  to  jettison  or  to  land? 

4 

11.  If  one  engine  is  not  sufficient,  what  is  usually  the  reason? 

Load?  N 

12.  (If  Yes)  In  that  case,  would  you  always  try  to  jettison  first, 
and  then  decide  whether  one  engine  is  sufficient  to  continue? 

13.  In  general,  what  makes  you  decide  whether  to  jettison  or  not? 

14.  Does  the  safety  of  the  landing  site  make  a difference?  ' 

15.  How  do  you  decide  whether  it  is  better  to  land  and  abort  the  mission, 
or  to  jettison  and  continue? 

16.  Can  you  generally  place  a monetary  value  on  the  loss  incurred  by 
jettisoning? 

17.  Is  it  generally  the  same  each  time?  ( i . e . , you  tend  to  get  rid 
of  the  same  things?) 

18.  (Depending  on  the  cost  involved  in  jettisoning)  would  you  try  to 
land  in  an  unsafe  landing  site  rather  than  to  jettison? 

19.  Of  different  possibilities  with  single  engine  failure,  one  engine 
not  sufficient,  and  unsafe  landing  site; 

(a)  Which  will  you  try  first?  Why? 

(b)  What  does  it  depend  on? 

t 

(c)  Are  the  environmental  conditions  more  important  or  the  possible 
losses  that  you  might  incur?  * 
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20.  At  what  point  do  you  start  thinking  about  bailing  out? 

21.  You  have  attempted  to  jettison,  one  engine  is  still  not  sufficient, 
and  you  can't  find  a safe  landing  site,  which  will  you  do  first: 

(1)  try  to  land  anyway,  (2)  bail  out,  or  (3)  attempt  restart? 

What  does  it  depend  on? 

22.  (Same  question  following  "attempt  restart"  and  failure:  land  or 
bail  out?) 

Dual  Engine  Failure  - Over  Land  1 

Discuss  the  possibilities  as  above: 

What  do  you  do  first?  Why?  Are  there  other  possibilities? 

Do  you  always  do  the  same  thing  first?  What  does  it  depend  on? 

If  you  have  a small  chance  of  landing  safely  by  auto-rotating,  do  you 
take  it?  (rather  than  lose  the  helicopter?). 

Over  Water 

Questions  concerning  the  importance  of  the  mission  and  the  losses  involved 
in  jettisoning  apply  here. 

Land  or  Wave-Off 


1.  What  is  your  first  consideration  when  deciding  whether  to  land  or 
wave  off?  Could  you  rank  the  following  in  terms  of  their  importance 
in  your  decision? 
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(a)  Weight  of  cargo 

(b)  Environmental  conditions 

(c)  Landing  site  (prepared/unprepared) 

(d)  Speed  of  helicopter 

(e)  Sink  rate 

(f)  Fuel  amount 

2.  Are  there  other  considerations? 

3.  How  do  you  decide  whether  a particular  situation  is  safe  or  not? 
Could  you  look  at  all  the  outcomes  and  rate  them  in  terms  of  the 
dangers  involved?  For  example,  you  have  a heavy  cargo,  the 
environmental  conditions  are  good,  and  you  have  decided  on  a 
running  landing,  but  you  are  going  too  fast,  would  you  try  to 
land,  or  would  you  wave  off?  If  you  try  to  land  (because  of 
low  fuel)  is  it  safer  (or  less  dangerous)  than  if  you  tried  to 
land  under  the  same  conditions  but  with  a high  sink  rate  (rather 
than  with  high  speed)? 

(a)  If  you  have  plenty  of  fuel,  would  you  always  wave-off  on 
the  first  approach: 

If  you  are  going  too  fast? 

If  you  have  too  high  a sink  rate? 

(b)  Does  each  wave-off  "cost"  something?  That  is,  aside  from 
using  up  fuel,  are  there  other  dangers  involved  in  waving 
off? 


How  would  you  rate  these  dangers? 
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(c)  Are  your  decisions  different  (re-land  or  wave-off)  depending 
on  whether  you  are  planning  a hover  - or  a running  - 
> landing? 

How?  Why? 

Ditching 

1.  How  many  different  types  of  alerts  are  there  that  suggest  a ditching 
situation? 

2.  How  different  are  they  from  each  other? 

3.  Can  you  always  tell  whether  the  alert  suggests  an  emergency  versus 
a planned  ditching?  In  general,  how  often  does  each  happen? 

4.  If  the  alert  suggests  a planned  ditching,  in  general,  how  long  do 
you  have  until  you  have  to  make  a decision? 

5.  Do  you  start  looking  for  a landing  site  as  soon  as  you  become 
aware  of  the  alert? 

6.  If  you  are  over  land,  do  you  try  to  land  as  soon  as  possible? 

7.  If  you  are  over  water,  how  long  do  you  wait  until  you  decide  to 
ditch? 

8.  When  you  first  become  aware  of  the  alert,  is  there  a way  to  estimate 
how  likely  it  is  that  it  is  a correct  (or  an  incorrect)  alert? 
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9.  Do  the  "alert"  cues  change  over  time,  so  that,  for  example,  you  may 
first  decide  to  look  for  a proper  landing  site  (over  land,  during 
planned  ditching),  and  then  change  your  mind  and  decide  to  land 
as  soon  as  possible? 

10.  If  so,  what  are  the  changes  in  the  alert  condition  that  produce  this 
switch?  Are  they  clear-cut  changes  or  is  it  a matter  of  "judgment"? 

11.  If  it  is  a matter  of  judgment,  can  you  elaborate? 

12.  Do  you  consider  what  it  will  cost  if  you  ditch?  To  you?  To  the 
Navy? 
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APPENDIX  I 

TRIP  TO  LOS  ANGELES  POLICE  ACADEMY 

The  Los  Angeles  Police  Academy  in  Pasadena  has  recently  constructed 
a simulator  for  training  decisions  in  "shoot/no  shoot"  situations.  That 
is,  the  simulator  focuses  on  the  problem  of  when  to  shoot  rather  than 
how  to  shoot  or  at  what  target.  The  simulator  is  built  in  a one-room, 
two-story,  cubic  building.  A giant  movie  screen  fills  one  entire  wall 
from  floor  to  ceiling  with  movie  projectors  located  in  a small  booth 
near  the  ceiling  on  the  opposite  wall.  A quadraphonic  sound  system 
accompanies  the  films. 

The  police  cadet  stands  in  front  of  the  screen  with  a special 
laser  pistol  that  resembles  his  service  revolver.  The  filmed  scenario 
places  the  cadet  in  a situation  in  which  he  must  decide  whether  or  not 
to  shoot  at  a criminal  who  is  threatening  his  life.  Two  video  cameras 
monitor  the  cadet  during  the  scenario.  One  is  located  near  the  movie 
projector  and  the  other  is  mounted  directly  above  the  screen  itself  in 
order  to  provide  a front  view. 

As  the  cadet  shoots,  a light-sensing  device  located  in  the  rear 
of  the  room  pinpoints  the  location  of  the  "shot"  on  the  screen  to  an 
accuracy  of  one  inch.  This  information,  plus  the  exact  movie  frame  that 
was  showing  at  the  time,  are  input  to  a Nova  mini-computer  and  stored 
for  later  processing.  At  the  end  of  the  scenario,  a videotape  is  prepared 
using  "split  screen"  technique,  of  the  critical  portions  of  the  exercise. 
The  action  on  the  screen  is  placed  on  the  top  half  of  the  video  image 
and  the  cadet  is  placed  in  the  bottom  half.  This  tape  is  used  to 
review  the  cadet's  actions  at  a later  time.  By  using  both  the  front 
and  rear  cameras,  the  location  of  the  cadet  and  the  location  of  his 
shot  on  the  screen  can  be  determined  to  an  accuracy  of  one  inch  and 
within  1/24  of  a second. 
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Although  there  is  no  formal  training  in  either  probability  theory 
or  decision  theory,  there  are  extensive  courses  in  the  established 
important  attributes  that  should  be  considered  when  deciding  whether  or 
not  to  shoot  in  a given  situation.  Some  of  these  attributes  are: 

(1)  legal  aspects; 

(2)  moral  aspects;  and 

(3)  personal  safety,  safety  of  bystanders,  safety  of  fellow  officers. 

There  is,  thus,  no  relative  weighting  scheme  for  the  attributes 
nor  decision  rules  that  translate  the  utilities  of  the  attributes  into  a 
decision.  There  are,  however,  general  guidelines  that  help  the  cadets 
make  the  decision.  Since  the  shoot/no  shoot  decision  must  usually  be 
made  in  less  than  one  second,  there  is  little  or  no  time  to  consider 
attributes,  utilities,  or  probabilities  at  that  time.  Consequently,  the 
cadets  are  taught  to  prune  the  decision  tree  before  the  actual  situation 
arises.  The  more  aspects  considered  and  dealt  with  early,  the  less  to 
consider  when  the  time  comes  to  make  the  decision  itself. 
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APPENDIX  J 

DECISION  TASK  CLASSES 
SINGLE  ATTRIBUTE  DECISION  MAKING 

General  Definition:  A decision  where  the  action  taken  and  the 
state  of  the  world  in  combination  determines  the  outcome  (payoff)  where 
the  outcome  (payoff)  is  a single  value  dimension  (attribute). 

Amplification:  A decision  maker  is  frequently  asked  to  choose 
between  courses  of  action  whose  probable  consequences  are  characterized 
by  a single  value  attribute  (for  example,  deciding  on  the  number  of 
sonobuoys  of  a particular  type  to  take  on  a mission).  The  outcomes, 
depending  on  the  number  taken  and  the  number  of  submarine  contacts 
made  (state  of  nature)  will  determine  the  single  attributed  outcome: 
adequacy  of  sonobuoy  type  supply  on  hand.  Adequacy  of  supply  is  a single 
outcome  ranging  from  some  numerical  shortage  to  a numerical  surplus. 

Rul es : Identify  the  decision  outcome  for  each  action/state  of 
the  world.  The  outcome  may  be  either  estimated  directly  by  experts, 
predicted  by  modeling,  or  predicted  by  experimental  evidence.  Each 
outcome  may  be  either  wholly  subjective,  subjective  estimates  on  an 
objective  scale,  or  objective  measurements  on  an  objective  scale. 

Pitfalls  and  Limitations:  The  most  important  pitfall  is  the 
representation  of  an  outcome  by  a single  point  estimate  when  the  outcome 
itself  may  have  a range  or  may  be  a probability  density  function. 

Interactions  with  Other  Decision  Elements.  No  major  interactions. 

Prerequisites.  There  must  be  a method  or  mechanism  that  permits 
(trains  or  projects)  outcome  estimation. 
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MULTIPLE  ATTRIBUTE  DECISION  MAKING 

General  Definition:  A decision  situation  where  the  action  taken 
and  the  state  of  the  world  in  combination  determines  the  outcome  (payoff) 
where  the  outcome  (payoff)  involves  multiple  value  dimensions  (attributes). 

Amplifications:  Decision  makers  frequently  choose  between  courses 
of  action  whose  probable  consequences  are  each  characterized  by  multiple 
value  attributes.  For  example,  the  decision  of  whether  or  not  to  ditch 
a LAMPS  helicopter  involves  a number  of  value  relevant  factors  such  as 
worth  of  aircraft,  safety  to  individuals  in  the  helicopter,  safety  to 
those  on  the  ground,  adverse  reaction  if  it  is  shown  later  by  investigation 
that  the  ditch  was  unnecessary,  and  so  on.  In  making  a decision  in  such 
a situation,  all  such  value-relevant  factors  must  be  identified  for  each 
alternative/state  of  nature. 

Rules:  Identify,  for  each  action  alternative/state  of  nature, 
the  set  of  outcomes  across  all  value  dimensions.  Outcomes  may  be  either 
estimated  directly  by  experts,  predicted  by  modeling,  or  predicted  by 
experimental  evidence.  Each  outcome  may  be  either  wholly  subjective, 
subjective  estimates  on  an  objective  scale,  or  objective  measurement  on 
an  objective  scale. 

Pitfalls  and  Limitations:  The  most  important  pitfall  is  the 
representation  of  an  outcome  by  a point  estimate  when  in  fact  the  outcome 
is  a range  or  may  itself  be  a probability  density  function. 

Interactions  with  Other  Decision  Elements:  No  major  interactions. 

Prerequisites:  There  must  be  a method  or  mechanism  for  generating 
outcome  estimates.  This  may  involve  training  outcome  estimation,  providing 
standard  lists  of  outcome  estimation,  and  so  on. 
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WELL  DEFINED  DECISION  MAKING  SITUATIONS 

General  Definition.  A decision  situation  where  the  action 
alternatives,  states  of  nature,  and  outcome  set  is  completely  describable, 
well  defined,  well  understood  in  advance  of  the  actual  decision  making 
situation. 

Amp! ification.  A key  to  any  decision  situation  is  problem 
structuring.  The  prerequisites  for  structuring  are:  (1)  knowledge  of 
the  complete  set  of  action  alternatives  to  be  considered,  and  (2) 
knowledge  of  all  likely  states  of  nature  that  will  be  involved  in  the 
situation.  If  these  prerequisites  are  met,  the  situation  is  well  defined 
and  can  easily  be  structured.  For  example,  in  the  event  of  engine 
failure  in  the  LAMPS  helicopter,  the  craft  may  be  over  land  or  over  water 
and  the  failure  may  be  due  to  fuel  starvation,  dual  engine  failure,  or 
single  engine  failure.  The  alternatives  are  clear  if,  for  example,  single 
engine  failure  occurs:  the  pilot  may  land,  continue,  abort  the  mission, 
or  look  for  a safe  site  if  the  aircraft  is  over  an  unsafe  landing  site 
(state  of  nature).  In  this  example,  the  decision  situation  is  well  defined. 

Rules:  There  are  two  possible  mechanisms  used  to  present  the 
detailed  structure  of  well-defined  decision  making  situations:  (1)  the 
payoff  (or  loss)  table,  and  (2)  the  decision  tree.  The  general  procedure 
is  to  express  the  decision  alternatives  and,  for  each  alternative,  list 
consequences  under  each  state  of  nature.  In  one  case,  the  result  is  in 
matrix  form,  in  the  other  the  result  is  in  tree  diagram  form.  If  there 
are  a sequence  of  decisions,  a decision  tree  is  usually  best  for 
structuring. 

Pitfalls  and  Limitations:  A tree  or  matrix  summarizing  a decision 
situation  forces  closure  in  a specific  decision  situation.  There  is  a 
tendency  to  view  the  decision  in  terms  of  the  tree  or  matrix  only,  rather 
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than  realizing  that  the  actual  decision  may  involve  more  than  what  is  just 
shown  in  the  structure.  That  is,  the  closure  may  be  in  error  with  a 
major  element  of  the  decision  omitted. 

Interactions  with  Other  Decision  Elements:  The  outcomes  to  be 
displayed  are  either  single  or  multiple  and  are  generated  as  indicated 
earlier.  Also,  there  may  not  be  enough  time  to  create  a large  structure. 

Prerequisites:  A tree  or  matrix  structure  should  be  developed  or 
provided.  If  developed,  the  basic  material  for  constructing  such  a 
structure  must  be  provided.  The  set  of  possible  action  alternatives 
should  be  provided  with  the  outcomes  for  each  prespecified  alternative 
state  of  nature.  The  set  of  action  alternatives  should  be  mutually 
exclusive  and  exhaustive. 
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AMBIGUOUS  DECISION  MAKING  SITUATIONS 


General  Definition:  A decision  situation  where  the  action 
alternatives,  states  of  nature,  and  outcome  set  is  neither  well  defined 
nor  completely  understood  prior  to  or  during  the  actual  decision  making 
situation. 

Amplification:  The  prerequisite  for  structuring  any  decision 
situation  as  indicated  previously  are  (1)  knowledge  of  the  complete  set 
of  possible  action  alternatives  to  be  considered  in  the  decision 
situation  and  (2)  knowledge  of  all  likely  states  of  nature  that  will 
be  involved  in  the  situation.  In  ambiguous  situations,  either  one  or 
both  of  these  preconditions  for  well  defined  decision  making  structuring 
fail.  There  are  two  distinct  kinds  of  failure:  (1)  the  set  of  states 
of  nature  or  the  set  of  possible  action  alternatives  is  so  large  that 
for  practical  purposes  it  cannot  be  represented  in  matrices  or  tree 
diagrams,  (2)  the  set  of  states  of  nature  of  the  set  of  possible  action 
alternatives  cannot  be  anticipated  completely  in  advance  of  the  actual 
decision  situation.  For  example,  what  should  the  pilot  or  ATO  do  when 
the  LAMPS  looses  contact  with  a submarine.  While  the  general  set  of 
alternatives  and  states  of  nature  are  known,  the  complete  sets  are  not. 
In  any  given  situation,  there  may  be  alternatives  or  states  of  nature 
unique  to  that  situation. 


Rules : As  in  the  case  of  well  defined  situations,  structure 
what  is  known  in  matrices  or  decision  trees  and  specify  unknowns  where 
appropriate.  In  addition,  indicate  that  part  of  the  structuring  must 
occur  at  the  time  of  the  actual  decision.  The  concept  of  closure  must 
be  firmly  addressed  as  well  as  sufficiency  of  the  alternative  set. 
Sensitivity  analysis  should  be  conducted  to  determine  decision  sensitivity 
to  structure. 
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Pitfalls  and  Limitations:  In  particular,  any  structure  provided 
tends  to  take  on  more  value  than  is  warranted.  Preliminary  structuring  may 
preclude  additional  structuring  as  an  expedience.  There  are  no  complete 
rules  for  specifying  when  all  the  likely  and  relevant  states  of  nature 
have  been  specified. 

Interactions  with  Other  Decision  Elements:  The  outcomes  to  be 
displayed  are  either  single  or  multiple  and  are  generated  as  indicated 
earlier.  Also  there  may  not  be  enough  time  to  create  a large  structure. 

Prerequisites : A tree  or  matrix  should  be  developed  or  provided. 
Particular  attention  should  be  given  to  providing  methods  for  forcing 
consideration  of  additional  states  of  nature  and  alternative  combinations. 
The  set  of  alternatives  and  states  of  nature  provided,  while  not  exhaustive, 
must  be  mutually  exclusive. 
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TIME  RELAXED  DECISION  MAKING 

General  Definition:  A decision  situation  where  the  decision  maker 
has  sufficient  time  to  completely  structure  the  decision  in  terms  of  number 
of  attributes,  alternatives,  states  of  nature,  and  outcomes  as  well  as 
selecting  the  best  alternative.  In  short,  there  are  no  time  constraints. 

Amp! ification.  The  decision  making  situations  where  a decision 
maker  has  literally  "plenty"  of  time  to  structure  and  make  a decision 
represent  time  relaxed  decision  making.  For  example,  during  pre-mission 
planning  in  LAMPS  operations,  there  is  often  enough  time  for  discussion 
and  planning. 

Rules:  There  are  no  specific  rules  except  to  be  thorough  in  all 
aspects  of  decision  structuring  and  alternative  selection.  Moreover, 
there  is  enough  time  to  conduct  sensitivity  analyses  on  alternative 
selection  and  decision  structuring  as  well  as  the  quantification  process 
itself. 

Pitfalls  and  Limitations.  With  sufficient  time  to  make  a decision, 
decision  makers  often  postpone  the  decision  until  it  is  no  longer  time 
relaxed.  There  is  also  a tendency  to  become  "sloppy"  if  there  is  too  much 
time.  Moreover,  if  there  is  too  much  lead  time,  the  decision  situation 
may  change  substantially  by  the  time  the  actual  decision  is  to  be  made, 
thereby,  negating  all  the  decision  making  efforts.  There  is  sometimes  a 
temptation  to  do  something,  anything,  rather  than  wait  to  take  action. 

Interactions  with  Other  Decision  Elements.  No  major  interactions. 

Prerequisites . None . 


AD-A060  025 


UNCLASSIFIED 


PERCEPTION ICS  INC  «OOOLANO  HILLS  CALIF  f/Q  5/9 

ANALYSIS  OF  REQUIREMENTS  AND  METhOOOlOGY  FOR  DECISION  TRAINING  — ETC(U) 
FEB  76  J SALEH,  A LEAL.  L LUCACCINI  N61339-77-C-0005 


PfTR-10%4-76-2 


NAVTRAEQOIPC-77-C-0005-1  NL 


3<*3 

*86002  3 

a 

9 

p 

3 

B 

9 

9 

9 

g 

9 

1 

B 

g 

0] 

END 

DATE 

FILMED 

! 2-78 

DOC 

NAVTRAEQUIPCEN  77-C-0005-1 


TIME  PRESSURE  DECISION  MAKING 

General  Definition.  A decision  situation  where  the  decision  maker 
has  insufficient  time  to  completely  structure  the  decision  in  terms  of 
number  of  attributes,  alternatives,  states  of  nature,  and  outcome  as  well 
as  alternative  selection.  In  summary  there  are  time  constraints  and  the 
greater  the  constraints,  the  greater  the  pressure. 

Amplification.  In  those  situations  where  fast  moving  events 
preclude  adequate  consideration  of  structuring  a decision  and  selecting 
an  alternative,  there  is  time  pressured  decision  making.  For  example, 
when  an  engine  quits  in  the  LAMPS  helicopter  due  to  an  engine  fire,  there 
is  very  little  time  to  reflect  on  the  alternative  courses  of  action, 
their  potential  outcomes  three  or  more  steps  ahead,  to  predict  the  various 
relevant  states  of  nature,  and  then  to  select  a decision  rule  and  use  it 
to  select  an  optimal  alternative  course  of  action. 

Rules:  There  are  only  rules  of  thumb  under  time  pressure.  The 
most  obvious  one  is  "Don't  take  longer  than  the  time  available  to  structure 
the  decision  and  select  an  action."  If  this  happens,  the  decision  is  out 
of  the  decision  maker's  control.  The  general  rule  is  to  spend  time  on 
structuring  the  decision  at  the  expense  of  a hasty  action  selection.  In 
most  cases,  the  wrong  decision  following  a correct  structuring  is  easier 
to  recover  from  than  the  correct  decision  on  the  wrong  problem.  Also,  try 
to  focus  on  the  major  decision  elements  immediately,  foregoing  a close 
look  at  the  secondary  considerations. 

Pitfalls  and  Limitations:  The  idea  of  the  optimal  decision  under 
time  pressure  is  illusory.  The  best  that  one  can  hope  for  in  many  instances 
is  to  avoid  making  a "big"  mistake  by  utilizing  the  knowledge  provided  in 
previous  trainings.  Decision  analysis  is  thought  to  be  worse  than  making 
some  (or  any)  decision. 
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Interaction  with  Other  Decision  Elements:  Time  pressure  normally 
effects  all  other  decision  elements  adversely. 

Prerequisites:  Training  in  decision  making  under  pressure  can  help 
enormously. 
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STATIC  DECISION  MAKING 

General  Definition.  A decision  situation  is  static  if  the 
consequences  of  the  decision  specified  through  the  outcome  set  are  constant 
over  time  (i.e.,  not  a function  of  time). 

Amplification.  Consider  the  time  horizon  over  which  the 
consequences  of  a decision  unfold.  If  the  consequences  (outcome  set) 
remain  constant  over  that  time  horizon,  the  decision  is  static.  This 
issue  is  critical  since  the  outcome  set  is  estimated  in  advance  of  the 
decision  and  correct  representation  is  required  for  properly  structuring 
the  decision.  For  example,  in  deciding  about  the  number  of  sonobuoys  to 
drop  in  a particular  pattern,  we  consider  that  each  number  of  sonobuoys 
will  generate  a specific  "listening"  coverage  on  the  ocean  surface.  This 
listening  coverage  remains  constant  over  the  time  horizon  of  the  decision 
which,  in  this  example,  is  the  lifetime  of  the  particular  sonobuoys  used. 

Rules:  Determine  the  pattern  of  decision  consequences  (outcome) 
over  time  and  if  the  response  pattern  is  constant,  the  situation  is 
static  and  a point  estimate  may  be  used  to  represent  the  outcome.  In 
essence,  a "snapshot"  of  the  consequence  may  be  taken  at  any  point  in 
time  to  represent  the  outcome. 

Pitfalls  and  Limitations.  In  some  instances,  there  will  be 
outcomes  that  are  both  static  and  dynamic  (vary  with  time)  and  they 
must  be  identified  and  treated  separately.  Confusion  can  be  minimized 
if  the  consequence  magnitude  is  plotted  over  time  to  determine  if 
magnitude  is  a function  of  time. 

Interactions  with  Other  Decision  Elements.  There  is  some 
interaction  with  outcome  estimation  that  follows  from  attribute  (value 
dimension)  specification.  In  some  instances,  dynamic  patterns  can  be 
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represented  by  static  attributes  by  careful  wording  in  identifying  the 
static  attribute.  For  example,  peak  response  takes  a characteristic  of 
a dynamic  response  and  represents  it  by  a static  attribute. 

Prerequisites:  An  understanding  of  how  behavior  varies  as  a function 

of  time. 
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DYNAMIC  DECISION  MAKING 

General  Definition:  A decision  situation  where  the  action  taken 
produces  consequences  that  vary  as  a function  of  time. 

Amplification:  Consider  the  time  horizon  over  which  the  consequences 
of  a decision  unfold.  If  the  consequences  (outcome  set)  are  not  constant 
(i.e.,  display  of  pattern  over  the  time  horizon)  the  decision  is  dynamic. 

This  has  critical  implications  for  outcome  specification  since  the  pattern 
cannot  be  captured  by  a single  number.  The  outcome,  in  fact,  may  have  a 
number  of  transient  states  and  an  eventual  equilibrium,  each  state  effecting 
the  decision  to  be  made  in  different  ways.  For  example,  the  ratio  of  the 
number  of  active  sonobuoys  to  passive  sonobuoys  may  vary  over  the  time 
horizon  of  a mission.  This  is  particularly  true  if  the  initial  ratio  of 
buoys  (active  to  passive)  is  "ideal"  for  one  particular  mission  scenario 
in  the  initial  phases,  but  becomes  worse  if  the  scenario  unfolds  later 
in  a manner  other  than  anticipated.  The  operator  then  is  faced  with  a 
new  "situation"  and  may  have  to  reorient  his  decision  attitude.  In 
the  event  of  engine  failure,  the  "dangerousness"  of  the  decision  outcome 
will  change  as  the  craft  passes  from  over  water  to  over  land.  If  the 
initial  failure  occurs  over  water  and  the  pilot  decides  to  try  and  make 
it  home  on  one  engine,  the  decision  outcome  is  at  one  "level  of  danger" 
over  water  and  another,  lower  "level  of  danger"  over  land.  Care  must  be 
given  to  using  the  respective  durations  in  trying  to  develop  numbers  to 
represent  the  decision  outcomes,  not  just  a single  point  estimate. 

Rules:  Determine  the  pattern  of  decision  consequences  (outcomes) 
over  time  and  if  the  response  pattern  changes  over-time,  the  situation 
is  dynamic  and  the  entire  response  pattern  and  its  characteristics  must 
be  considered  In  order  to  represent  the  outcome.  Characteristics  of 
response  patterns  such  as  peak  and  valley  magnitudes,  durations  of  peaks 


196 


NAVTRAEQUIPCEN  77-C-0005-1 


and  valley,  transients,  ultimate  steady  state,  time  to  reach  steady 
state,  and  so  on,  must  be  considered. 

Pitfalls  and  Limitations:  In  some  instances  the  decision  between 
static  and  dynamic  will  be  confounded  by  the  distinction  between  discrete 
and  continuous  decision  making. 

Interactions  with  Other  Decision  Elements:  See  comments  under 
static  decision  making.  In  addition,  dynamic  consequences  represented  by 
static  attributes  tend  to  make  single  attribute  decision  making  into 
multi-attribute  decision  making.  Hence,  dynamic  outcomes  may  preclude 
single  attribute  decision  making. 

Prerequisites:  An  understanding  of  how  behavior  varies  as  a 
function  of  time,  and  ability  to  characterize  components  of  patterns  of 
behavior  such  as  durations,  peaks,  periodicity,  etc.,  are  important 
prerequisites. 
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SUBJECTIVE  EXPECTED  UTILITY  (SEU) 

4 

General  Definition:  The  alternative  selection  rule  where  the 
alternative  with  the  highest  subjective  expected  utility  is  selected. 

Ampl ification.  For  each  outcome  and  state  of  nature  developed 
in  the  problem  structuring  phase,  determine  the  utility  of  the  outcome(s) 
and  the  probabilities  of  the  various  states  of  nature.  The  decision 
tree  is  then  "rolled  back"  and  the  alternative  with  the  highest  SEU 
is  chosen. 

\ 

Rules:  There  are  a number  of  ways  of  eliciting  utilities  and 
determining  the  relevant  probabilities.  Direct  estimation  is  often  the 
most  useful  in  field  settings  and  the  results  correlate  closely  with 
alternative  (and  more  complicated)  elicitation  techniques.  When  performing 
"rollback"  on  a decision  tree,  at  each  decision  node,  take  the  maximum 
of  the  successor  values  and  at  each  event  node,  take  the  expected  value 
of  the  successor  values. 

Pitfalls  and  Limitations:  The  problem  with  SEU  is  that  is  does 
not  protect  the  decision  maker  from  the  "worst  case"  event  in  nature. 

The  alternative  chosen  has  the  highest  expected  utility  but  in  retrospect, 
a "good"  decision  may  have  produced  a "bad"  outcome  if  nature  happened 
to  be  against  the  decision  maker. 

Interaction  with  Other  Decision  Elements:  Under  time  relaxed, 
no  significant  interactions.  Under  time  pressure,  the  decision  maker 
may  have  to  make  more  global  estimates  closer  to  the  beginning  of  the 
decision  tree  rather  than  at  the  more  operational  level. 

Prerequisites:  Training  in  utility  assessment,  aggregation, 
probability  estimation,  and  decision  tree  rollback  are  desired. 

Q 
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MAXI-MIN  UTILITY 

General  Description:  A payoff  table  is  constructed  (with  any 
multiattribute  outcomes  aggregated  to  a single  utility)  and  expressed 
in  utilities,  then  the  alternative  is  chosen  that  maximizes  the  minimum 
utility. 

Amplification:  Each  alternative  is  arrayed  for  each  state  of 
nature.  The  outcomes  are  all  expressed  in  utilities.  Since  nothing 
is  known  or  assumed  (or  estimated)  about  the  probabilities  of  the  various 
states  of  nature,  each  alternative  is  examined  to  find  the  worst  outcome. 
When  the  worst  case  utility  is  identified  for  each  alternative,  the 
alternative  with  maximum  "worst"  utility  is  chosen.  Therefore,  no 
matter  what  state  of  nature  eventually  occurs,  the  utility  of  the 
outcomes  can  be  no  less  than  the  minimum  utility  of  the  alternative 
chosen.  The  decision  rule  guarantees  the  best  of  the  worst,  or  maximizes 
the  minimum  utility.  Hence,  a pessimistic  decision  maker  is  always 
covered  against  the  worst  case. 

Rules:  Construct  the  payoff  matrix  and  estimate  the  utilities 
of  each  alternative  under  each  state  of  nature.  Then  identify  the  lowest 
utility  for  each  alternative  under  any  state  of  nature  for  each 
alternative  and  select  the  alternative  with  the  highest  utility. 

Pitfalls  and  Limitations:  The  method  assumes  that  nothing  is 
known  about  the  relative  probabilities  of  the  states  of  nature  and, 
hence,  assumes  that  each  are  equally  likely.  This  may  not  be  so  in 
a given  decision  situation.  Also,  the  method  does  not  lead  to  an 
optimal  decision  in  the  expected  utility  sense. 

Prerequisites:  Training  in  utility  estimation  and  aggregation 
and  payoff  table  construction  are  prerequisites. 
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MAX  I -MAX  UTILITY 

General  Description:  A payoff  table  is  constructed  with  all 
outcomes  (single  and  multiattribute)  represented  by  a single  utility 
and  then  the  alternative  that  maximizes  the  maximum  utility  is  chosen. 

Amplification:  Each  alternative  is  arrayed  for  each  state  of 
nature  and  the  outcomes  are  all  expressed  in  utilities.  Since  nothing 
is  known,  assumed,  or  estimated  about  the  probabilities  of  the  various 
states  of  nature,  each  alternative  is  examined  to  find  the  best  outcome 
(i.e.,  the  maximum  utility).  Then  the  highest  utility  is  identified 
for  each  alternative  (regardless  of  the  state  of  nature  under  which 
it  occurs)  the  alternative  with  the  maximum,  highest  utility  is  chosen. 
This  decision  rule  optimistically  assumes  that  if  the  best  happens,  the 
alternative  will  have  been  chosen  to  deliver  the  maximum  utility. 

Rules:  Construct  the  payoff  matrix  and  estimate  the  utilities 
of  each  alternative  under  each  state  of  nature.  Then  identify  the 
highest  utility  for  each  alternative  regardless  of  the  state  of  nature 
under  which  it  occurs,  and  select  the  alternative  with  the  maximum 
(highest)  utility. 


Pitfalls  and  Limitations:  This  method  again  assumes  that  nothing 
is  known  about  the  probabilities  associated  with  the  various  states  of 
nature,  and  this  may  not  be  so.  Also,  this  method  may  reward 
optimists,  but  there  is  no  guarantee  against  losses  or  even  that  the 

optimal  alternative  is  chosen  in  a SEU  sense. 

• - f 

% ^ 

Interaction  with  Other  Decision  Elements:  Under  time  relaxed, 
no  interactions.  Under  time  pressure,  this  method  has  similar 
advantages  computationally  to  those  of  maxi-min  utility. 
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Prerequisi tes : Training  is  utility  assessment  and  aggregation  and 
in  payoff  table  construction. 


201 


NAVTRAEQU I PCEN  77-C-0005-1 


MINI-MAX  REGRET 

General  Description:  A pe.  off  table  is  constructed  and  from  it, 
a regret  matrix  is  constructed  showing  how  much  "regret"  the  decision 
maker  would  have  if  he  made  the  wrong  decision.  The  alternative  is 
chosen  that  minimizes  the  maximum  regret  of  the  decision  maker. 

Amplification:  Each  alternative  is  arrayed  in  a payoff  table  as 
in  maxi-min  utility.  A regret  matrix  is  constructed  by  finding  the  best 
possible  alternative  action  under  any  state  of  nature.  If  that  state  of 
nature  turned  up,  the  decision  maker  would  have  no  regret.  If  he  had 
chosen  any  other  alternative  under  that  state  of  nature,  he  would  regret 
it  by  the  amount  of  the  difference  between  the  best  he  could  do  and  the 
amount  for  the  chosen  alternative.  A regret  matrix  is  constructed 
comparing  how  much  a decision  maker  would  regret  his  decision  if  the  best 
occurred  under  each  state  of  nature  (i.e.,  he  had  chosen  the  best  alternative 
for  state  of  nature).  The  regrets  for  each  alternative  under  each  state 
of  nature  form  the  regret  matrix.  Each  alternative  is  scanned  to  find 
the  maximum  regret.  The  alternative  with  the  smallest  maximum  regret 
is  selected. 

Rules:  Construct  the  payoff  table.  Then,  construct  the  regret 
matrix,  identify  the  maximum  regrets,  and  select  the  alternative  with 
smallest  maximum  regret. 

Pitfalls  and  Limitations:  In  this  method,  the  addition  of  a new 
• alternative  that  itself  is  not  eventually  chosen,  can  shift  the  decision 

to  a different  alternative  other  than  the  one  chosen  before  the  addition. 
Also,  the  method  assumes  that  nothing  is  known  about  probabilities  of 
states  of  nature. 
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Interaction  with  Other  Decision  Elements:  Under  time  relaxed,  no 
significant  interactions.  Under  time  pressure  the  computations  are  time 
consuming. 

Prerequisi tes:  Training  in  utility  assessment  and  aggregation  and 
in  payoff  and  regret  matrix  construction. 
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LEXICOGRAPHY 

General  Description:  Lexicography,  as  used  here,  is  a procedure 
whereby  the  most  important  attribute  of  a decision  situation  is  considered 
for  each  alternative  and  the  decision  is  made  on  the  basis  of  which 
alternative  scores  the  highest  in  this  attribute.  In  the  event  of  a tie 
on  the  most  important,  the  next  most  important  attribute  is  identified 
and  the  alternative  that  scores  the  highest  on  it  is  selected.  In  the 
event  of  a tie  the  third  most  important  attribute  is  identified  and  the 
process  repeats  until  an  alternative  is  chosen.  Only  those  alternatives 
that  tie  are  considered  in  subsequent  rounds. 

Amplification:  Lexicography  is  like  looking  up  a word  in  the 
dictionary.  You  start  with  the  first  letter,  then  the  second  and  so  on 
until  you  have  found  the  word  you  are  looking  for.  Similarly  in 
lexicographical  decision  making  you  identify  the  most  important  attribute, 
and  then  make  a decision  on  it  unless  there  is  a tie.  In  the  event  of  a 
tie  the  process  repeats  on  the  second  most  important  then  the  third, 
and  so  on.  The  process  works  when  there  are  multi  attribute  alternatives. 
For  example,  in  the  event  of  an  engine  fire  in  a LAMPS  helicopter,  the 
pilot  must-cons ider  safety,  mission  completion,  and  so  on  in  deciding 
whether  to  ditch,  or  attempt  to  restart.  If  safety  is  the  most  important 
factor,  the  decision  would  be  made  by  selecting  the  alternative 
(ditching  or  restart  in  this  example)  that  scores  the  highest  on  this 
attribute.  If  the  decision  cannot  be  made  on  the  basis  of  safety  alone, 
mission  completion  would  then  be  considered,  etc. 

Rules:  Identify  the  relative  importance  of  each  attribute. 

Follow  the  lexicographical  procedures. 
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Pitfalls  and  Limitations:  A serious  limitation  in  this  technique 
is  that  the  second  and  third  most  important  attributes  together  may 
outweigh  the  first  most  important  one.  Hence,  clearly  suboptimal 
alternatives  may  be  selected.  This  method  assumes  that  only  one  dimension 
at  a time  is  relevant  for  decision  making,  and  that  it  is  more  important 
than  all  the  others. 

Interaction  with  Other  Decision  Elements:  Under  time  relaxed, 
no  significant  interactions.  Under  time  pressure  the  computations  are 
so  easy  that  there  are  no  significant  interactions. 

Prerequisites:  Training  is  establishing  the  relative  importance 
of  the  attribute  dimensions  in  mul tiattributed  decision  alternatives. 
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HURWICZ 

General  Description:  A payoff  table  is  constructed  so  that  each 
alternative  is  represented  by  a single  utility  number  of  each  state  of 
nature.  Then,  for  each  alternative  the  best  and  worse  (highest  and  lowest) 
utilities  are  identified.  This  represents  the  best  and  worst  case  (most 
optimistic  and  most  pessimistic).  The  alternative  that  has  the  best 
"combination"  of  best  and  worst  utilities  is  selected. 

Amplification:  The  attempt  here  is  to  combine  a pessimistic  and 
optimistic  approach.  The  decision  maker  establishes  how  likely  the  best 
and  worst  cases  are  to  occur  (by  establishing  probabilities  that  sum  to 
one  for  both).  The  expected  utility  is  then  calculated  using  only  the 
best  and  worst  utilities  and  their  respective  probability  estimates. 

Rules:  Establish  the  payoff  table  and  identify  the  best  and  worst 
utility  for  each  outcome.  Estimate  the  probabilities  for  each  (sum  to 
one).  Then  calculate  the  expected  utility  for  each  alternative. 

Pitfalls  and  Limitations:  This  approach  examines  only  the  best 
and  worst.  The  other  outcomes  are  ignored.  Also,  the  estimate  of  the 
probabilities  of  each  of  the  best  and  worst  occurring  is  subject  to  error. 

Interaction  with  Other  Decision  Elements:  Under  time  relaxed,  no 
interactions.  Under  time  pressure,  there  may  not  be  time  to  properly 
measure  the  probabilities. 

Prerequisites:  Constructing  payoff  table.  Assigning  and 
aggregating  utilities.  Estimating  coefficients  of  optimism  *nd 
pessimism  (the  probabilities). 


•■4 
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1 SATISFYING 

General  Description:  Establish  the  feasible  domain  for  all 
i>  alternatives.  Select  the  first  alternative  screened  that  falls  into 

the  feasible  domain. 

. 

Amplification:  For  multi -attribute  alternatives,  establish  the 
minimum  (or  maximum)  acceptable  boundaries  of  performance  for  each 
attribute.  This  defines  the  feasible  domain.  Any  alternative  screened 
that  falls  outside  of  these  limits  on  any  attribute  dimensions  is 
eliminated  from  further  consideration.  The  first  alternative  that 
meets  all  the  limits  stated  is  selected  without  further  consideration 
of  any  additional  alternatives. 

Rules:  Establish  the  minimum  and/or  maximum  thresholds  for  each 
attribute  value  dimension.  Plot  each  alternative  considered  on  the 
feasible  domain  and  select  the  first  to  fall  completely  within  the 
feasible  domain. 

Pitfalls  and  Limitations:  The  method  produces  an  alternative,  but 
it  may  be  the  worst  of  all  possible  alternative  that  are  feasible,  and 
hence  far  from  optimal . 

Interaction  with  Other  Decision  Elements:  Under  time  relaxed,  no 
interactions.  Under  time  pressure,  if  the  feasible  domain  boundaries 
can  be  established  in  time,  the  method  will  produce  a rapid  decision 
1 on  an  alternative  that  is  feasible  and  meets  all  minimums  (maximums) 

required. 

• 

Prerequisites:  Training  at  establishing  standard  minimums  and 
maximums  for  alternative  dimensions. 
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